Image analysis for determining characteristics of groups of individuals

ABSTRACT

Embodiments include methods for predicting one or more characteristics of an individual, such as a human or non-human animal, by applying computational methods to image(s) of the individual to generate one or more metrics indicative of the characteristics. Embodiments determine predictors of characteristics by creating a sample library of individuals, determining facial measurements for each individual, determining relationships between facial measurements and additional library data, and selecting predictors from these relationships. Embodiments include methods for predicting characteristics of individuals not in the library. Embodiments include methods for predicting characteristics of groups using predicted characteristics of individuals. Embodiments determine suitability of a pair of individuals (from the same or different species) for a particular purpose, task, or relationship based on characteristics of individuals. Other embodiments determine the compatibility of an individual with a group of other individuals. Embodiments include systems, devices, and computer-readable media comprising one or more of these methods.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 13/672,612, filed on Nov. 8, 2012, which is acontinuation-in-part of U.S. application Ser. No. 13/467,869, entitled“Image Analysis for Determining Characteristics of Animals,” and acontinuation-in-part of U.S. application Ser. No. 13/467,889, entitled“Image Analysis for Determining Characteristics of Humans,” both ofwhich were filed May 9, 2012. U.S. application Ser. No. 13/672,612 alsoclaims priority to U.S. Provisional Application Ser. No. 61/484,126,entitled “Image Analysis for Determining Characteristics of Animals,”which was filed May 9, 2011 and U.S. Provisional Application Ser. No.61/616,234, entitled “Image Analysis for Determining Characteristics ofAnimals,” which was filed Mar. 27, 2012, the content of theseapplications are incorporated herein by reference.

This application incorporates by reference U.S. application Ser. No.13/672,627 entitled “Image Analysis for Determining Characteristics ofIndividuals” filed on Nov. 8, 2012 by the same inventor and U.S.application Ser. No. 13/672,657 entitled “Image Analysis for DeterminingCharacteristics of Pairs of Individuals” filed on Nov. 8, 2012 by thesame inventor.

TECHNICAL FIELD

The disclosure herein relates to the objective determination of acharacteristic of an individual, such as a human or non-human animal, byapplying computational methods to one or more images of the individualto generate one or more metrics indicative of the characteristic ofinterest. It also relates to the determining the degree to whichparticular groups of two or more individuals are suitable for aparticular purpose, task, or relationship.

BACKGROUND

Animal domestication can be thought of as developing a mutually usefulrelationship between animals and humans. Over the past 12,000 years,humans have learned to control their access to food and othernecessities of life by changing the behaviors and natures of wildanimals. All of today's domesticated animals—including dogs, cats,cattle, oxen, llamas, sheep, goats, camels, geese, horses, chickens,turkeys, and pigs—started out as wild animals but were changed over thecenturies and millennia into animals that are tamer, quieter, andgenerally more cognitively suited to a lifestyle of coexistence withhumans. Today people benefit from domesticated animal in many waysincluding keeping cattle in pens for access to milk and meat and forpulling plows, training dogs to be guardians and companions, teachinghorses to adapt to the plow or take a rider, and changing the lean,nasty wild boar into the fat, friendly pig.

When individuals are looking to breed animals, they look for certaintraits in purebred stock that are valued for a particular purpose, ormay intend to use some type of crossbreeding to produce a new type ofstock with different, and, it is presumed, superior abilities in a givenarea of endeavor. For example, to breed chickens, a typical breederintends to receive eggs, meat, and new, young birds for furtherreproduction. Thus, the breeder has to study different breeds and typesof chickens and analyze what can be expected from a certain set ofcharacteristics before he or she starts breeding them. On the otherhand, purebred breeding aims to establish and maintain stable traitsthat animals will pass to the next generation. By “breeding the best tothe best,” employing a certain degree of inbreeding, considerableculling, and selection for “superior” qualities, one could develop abloodline superior in certain respects to the original base stock.

As first noted by Charles Darwin, domesticated animals are known toshare a common set of physical characteristics, sometimes referred to asthe domestication phenotype. C. Darwin, THE VARIATION OF ANIMALS ANDPLANTS UNDER DOMESTICATION (2^(nd) ed.) (New York: D. Appleton & Co.,1883). They are often smaller, with floppier ears and curlier tails thantheir untamed ancestors. Their coats are sometimes spotted while theirwild ancestors' coats are solid. One long-term study demonstrating thisphenomenon has been ongoing since 1958 at the Institute of Cytology andGenetics in Novosibirsk, Russia. In this study, scientists havesuccessfully demonstrated that, through careful selective breeding fortamability, wild Siberian silver foxes acquire both the behavioral andappearance traits of domesticated dogs. See, e.g., L. Trut, Early CanidDomestication: The Fox Farm Experiment, 87 AMERICAN SCIENTIST 160-69(March-April 1999). This highly conserved combination of psychologicaland morphological changes during the process of domestication is seen tovarying degrees across a remarkably wide range of species, from horses,dogs, pigs, and cows to some non-mammals like chickens and even a fewfish. However, in no other species has this relationship betweenbehavior and anatomical features been more widely noted than in thehorse.

Relationships also exist in humans between physiological feature sets(i.e., phenotypes) and certain cognitive functions and/or personalitytraits. During progressive stages of human embryonic growth, developmentof the brain and face remains intimately connected through both geneticsignaling and biomechanical/biochemical mechanisms. The face developsfrom populations of cells originating from the early neural crest, withcells from the neural tube gradually shifting to form the prominences ofthe face. Differentiation of these early cells is closely regulatedthrough intricate genetic signaling mechanisms, with the brainessentially serving as the platform on which the face grows. As thesetwo structures continue to grow and develop during the later embryonicstages, their phenotypes remain closely linked as complex genetichierarchies regulate patterns of cross talk between molecules, cells,and tissues.

SUMMARY

Embodiments comprise a method for measuring an individual, such as ahuman or non-human animal, to determine one or more characteristics ofthe individual, comprising receiving one or more digital imagesrepresenting the individual, storing the images in a computer memory,adding a plurality of reference points to the stored digital images, andcomputing one or more metrics relating to the characteristic of theindividual using the reference points. Other embodiments comprise amethod for determining a characteristic of an individual based on a setof metrics related to the individual, comprising selecting one or moremetrics from the set of metrics, calculating a combined metric using theselected metrics, and determining the characteristic of the individualbased on the value of the combined metric. Other embodiments comprisecomputer systems that implement one or more of the above methods.

Other embodiments include a method for predicting a human characteristiccomprising, for each of a plurality of individuals, storing one or moredigital images representing the individual in a memory operablyconnected to a digital computer; annotating the one or more digitalimages with a plurality of reference points; associating at least oneother data value about the individual with the one or more digitalimages representing the individual; computing, with the digitalcomputer, one or more metrics using the plurality of reference points.The method further comprises selecting a combination of the one or moremetrics for predicting the human characteristic. In some embodiments,the selecting step further comprises determining one or morerelationships between the one or more metrics and the at least one otherdata value for the plurality of individuals and the combination isselected based on the one or more relationships. Other embodimentscomprise systems and computer-readable media embodying these methods.

Other embodiments include a method for determining a characteristic ofan individual human based on one or more metrics related to theindividual, comprising storing one or more digital images representingthe individual in a memory operably connected to a digital computer;annotating the one or more digital images with a plurality of referencepoints; computing, with the digital computer, the one or more metricsusing the plurality of reference points; and predicting thecharacteristic of the individual based on the one or more metrics. Insome embodiments, the predicting step further comprises computing acombined metric based on a predetermined function of the one or moremetrics and predicting the characteristic based on the combined metric.In some embodiments, predicting the one or more characteristics of theindividual based on the one or more metrics comprises computing one ormore indicators used for predicting the one or more characteristics.Depending on embodiment, the one or more indicators may comprise one ormore primary indicators, one or more secondary indicators, one or moretertiary indicators, and/or one or more suitability indicators. Otherembodiments comprise systems and computer-readable media embodying thesemethods.

Other embodiments include a method for predicting one or morecharacteristics of a group of two or more individuals comprising, foreach of the individuals within the group, storing one or more digitalimages representing the individual in a memory operably connected to adigital computer; annotating the one or more digital images with aplurality of reference points; computing, with the digital computer, oneor more metrics using the plurality of reference points; and predictingone or more characteristics of the individual based on the one or moremetrics. The method further comprises predicting the one or morecharacteristics of the group of individuals based on the predicted oneor more characteristics of the individual comprising the group. Someembodiments further comprise determining a strategy relating to thegroup of individuals based on the predicted one or more characteristicsof the group. Some embodiments further comprise computing one or morecombined metrics, each based on a predetermined function of the one ormore metrics, and predicting the one or more characteristics of anindividual based on the one or more combined metrics. In someembodiments, the group comprises a pair of individuals, and predictingthe one or more characteristics of the pair based on the one or moremetrics comprises computing one or more indicators used for predictingthe one or more characteristics. In some embodiments, the one or moreindicators may comprise one or more individual indicators (e.g.,principal, secondary, and/or tertiary indicators) and one or morepairwise indicators related to a pair of individuals. Other embodimentscomprise systems and computer-readable media embodying these methods.

Other embodiments include a method for determining a characteristic of asubject (e.g., a human), comprising calculating two or more ratios basedupon metrics related to a subject's head, wherein distances or anglesbetween reference points on the subject's head are used; predicting,using a computer and computations, a characteristic of the subjectwherein the two or more ratios are used and wherein data about a groupof subjects are referenced; and providing the predicted characteristicto an output device. Other embodiments comprise systems andcomputer-readable media embodying these methods.

Other embodiments include a method for choosing a combination of twoindividuals for a particular task, comprising computing one or moremetrics related to one individual; computing one or more metrics relatedto each of a plurality of second individuals; computing a combinationcharacteristic related to the combination of the first individual witheach of the plurality of second individuals, based on at least a portionof the one or more metrics related to the first individual and at leasta portion of the one or more metrics related to each of the plurality ofsecond individuals; and determining the combination of the firstindividual and one of the plurality of second individuals based on thecomputed combination characteristics. Depending on embodiment, the firstindividual may be a human or non-human animal and all of the pluralityof second individuals may be humans or members of the same non-humananimal species. Other embodiments comprise systems and computer-readablemedia embodying these methods.

DESCRIPTION OF THE DRAWINGS

The detailed description will refer to the following drawings, whereinlike numerals refer to like elements, and wherein:

FIG. 1A shows facial descriptor measurement Eye Height Proportion (EHP);

FIG. 1B shows facial descriptor measurement Eye Depth Proportion (EDP);

FIG. 1C shows facial descriptor measurement Maximal Eye Point Proportion(MaxEPP);

FIG. 1D shows facial descriptor measurement Minimal Eye Point Proportion(MinEPP);

FIG. 1E shows facial descriptor measurement Eye Height to LengthProportion (EHLP);

FIG. 1F shows facial descriptor measurement Eye Extrema Angle (EEA);

FIG. 1G shows facial descriptor measurement Upper Lateral Eye RoundnessProportion (ULERP);

FIG. 1H shows facial descriptor measurement Lower Lateral Eye RoundnessProportion (LLERP);

FIG. 1I shows facial descriptor measurement Upper Rostral Eye RoundnessProportion (URERP);

FIG. 1J shows facial descriptor measurement Lower Rostral Eye RoundnessProportion (LRERP);

FIG. 1K shows facial descriptor measurement Pupilary Area Ratio (PAR),which is the area the area covered by the pupil when maximally dilated,characterized by radially striated bands around the area when not fullydilated;

FIG. 1L shows facial descriptor measurement Eye Set Angle (ESA);

FIG. 1M shows facial descriptor measurement Eye Width Proportion (EWP);

FIG. 1N shows facial descriptor measurement Iris Boundary Proportion(IBP);

FIG. 1O shows facial descriptor measurement Caruncle Length Proportion(CLP);

FIG. 1P shows facial descriptor measurement Caruncle Angular Proportion(CAP);

FIG. 1Q shows facial descriptor measurement Upper Eye Angle (UEA);

FIG. 1R shows facial descriptor measurement Lower Eye Angle (LEA);

FIG. 1S shows facial descriptor measurement Front Eye Angle (FEA);

FIG. 1T shows facial descriptor measurement Eye Elevation Angle (EEA);

FIG. 1U shows facial descriptor measurement Eye Depression Angle (EDA);

FIG. 1V shows facial descriptor measurement Eye Set Depth Proportion(ESDP);

FIG. 2A shows facial descriptor measurement Maxillary Palate Angle(MPA);

FIG. 2B shows facial descriptor measurement Maxillary Plate ProminenceProportion (MPPP);

FIG. 2C shows facial descriptor measurement Facial Length Proportion(FLP);

FIG. 2D shows facial descriptor measurement Mid-Face Proportion (MFP);

FIG. 2E shows facial descriptor measurement Mandible Width Proportion(MWP);

FIG. 2F shows facial descriptor measurement Labio-Orbital Angle (LOA);

FIG. 2G shows facial descriptor measurement Naso-Orbital Angle (NOA);

FIG. 2H shows facial descriptor measurement Naso-Chilial Angle (NCA);

FIG. 3A shows facial descriptor measurement Upper Lip ProtuberanceProportion (ULPP);

FIG. 3B shows facial descriptor measurement Lower Lip ProtuberanceProportion (LLPP);

FIG. 3C shows facial descriptor measurement Upper Lip ThicknessProportion (ULTP);

FIG. 3D shows facial descriptor measurement Lower Lip ThicknessProportion (LLTP);

FIG. 3E shows facial descriptor measurement Lower Lip Maxima PointProportion (LLMPP);

FIG. 3F shows facial descriptor measurement Upper Lip Maxima PointProportion (ULMPP);

FIG. 3G shows facial descriptor measurement Lower Lip Angle (LLA);

FIG. 3H shows facial descriptor measurement Upper Lip Angle (ULA);

FIG. 3I shows facial descriptor measurement Lower Lip RoundnessProportion (LLRP);

FIG. 3J shows facial descriptor measurement Upper Lip RoundnessProportion (ULRP);

FIG. 3K shows facial descriptor measurement Lip Convergence Angle (LCA);

FIG. 3L shows facial descriptor measurement Mouth-to-Pupil WidthProportion (MtPWP);

FIG. 3M shows facial descriptor measurement Mouth-to-Nostril WidthProportion (MtNWP);

FIG. 3N shows facial descriptor measurement Mouth Thickness Proportion(MTP);

FIG. 3O shows facial descriptor measurement Mouth Width Angle (MWA);

FIG. 3P shows facial descriptor measurement Upper Lip Length Proportion(ULLP);

FIG. 3Q shows facial descriptor measurement Lower Lip Length Proportion(LLLP);

FIG. 3R shows facial descriptor measurement Upper Labial ThicknessProportion (ULTP);

FIG. 3S shows facial descriptor measurement Lower Labial ThicknessProportion (LLTP);

FIG. 4A shows facial descriptor measurement Jowl Fullness Proportion(JFP);

FIG. 4B shows facial descriptor measurement Jowl Connective Proportion(JCP);

FIG. 4C shows facial descriptor measurement Jowl Thickness Angle (JTA);

FIG. 4D shows facial descriptor measurement Jowl Set Angle (JSA);

FIG. 4E shows facial descriptor measurement Lambiomental InflexionProportion (LIP);

FIG. 4F shows facial descriptor measurement Chin Roundness Proportion(CRP);

FIG. 4G shows facial descriptor measurement Chin Height Proportion(CHP);

FIG. 4H shows facial descriptor measurement Jowl Depth Angle (JDA);

FIG. 5A shows facial descriptor measurement Nasal Length Proportion(NLP);

FIG. 5B shows facial descriptor measurement Nasal Width Proportion(NWP);

FIG. 5C shows facial descriptor measurement Alar-to-Iris WidthProportion (AtIWP);

FIG. 5D shows facial descriptor measurement Radix Protrusion Proportion(RPP);

FIG. 5E shows facial descriptor measurement Radix Protrusion Angle(RPA);

FIG. 5F shows facial descriptor measurement Dorsum Rounding Proportion(DRP);

FIG. 5G shows facial descriptor measurement Dorsum Maximal ProtuberancePoint Proportion (DMPPP);

FIG. 5H shows facial descriptor measurement Nose Rounding Proportion(NRP);

FIG. 5I shows facial descriptor measurement Nostril Tip RoundingProportion (NTRP);

FIG. 5J shows facial descriptor measurement Nasal Tip Angle (NTA);

FIG. 5K shows facial descriptor measurement Nostril Height Proportion(NHP);

FIG. 5L shows facial descriptor measurement Nares Protrusion Proportion(NaPP);

FIG. 5M shows facial descriptor measurement Nares Length Proportion(NaLP);

FIG. 5N shows facial descriptor measurement Nares Inflexion Proportion(NIP);

FIG. 5O shows facial descriptor measurement Full Nose Angle (FNA);

FIG. 5P shows facial descriptor measurement Nasal Spine DeviationProportion (NSDP);

FIG. 5Q shows facial descriptor measurement Nasal Tip InflectionProportion (NTIP);

FIG. 5R shows facial descriptor measurement Alar Flare Proportion (AFP);

FIG. 5S shows facial descriptor measurement Nasal Bridge WidthProportion (NBWP);

FIG. 5T shows facial descriptor measurement Nasal Spine Angle (NSA);

FIG. 6A shows facial descriptor measurement Brow Fullness Proportion(BFP);

FIG. 6B shows facial descriptor measurement Forehead InflexionProportion (FIP);

FIG. 6C shows facial descriptor measurement Forehead Slope Proportion(FSP);

FIG. 6D shows facial descriptor measurement Forehead RoundnessProportion (FRP);

FIG. 6E shows facial descriptor measurement Forehead Maxima PointProportion (FMPP);

FIG. 6F shows facial descriptor measurement Forehead Height Proportion(FHP);

FIG. 6G shows facial descriptor measurement Eyebrow Height Proportion(EHP);

FIG. 6H shows facial descriptor measurement Forehead Width Proportion(FWP);

FIG. 6I shows facial descriptor measurement Forehead Fullness Proportion(FFP);

FIG. 6J shows facial descriptor measurement Hairline Inflection Angle(HIP);

FIG. 7A shows facial descriptor measurement Auricle Inclination Angle(AIA);

FIG. 7B shows facial descriptor measurement Auricle Set Angle (ASA);

FIG. 7C shows facial descriptor measurement Ear-to-Eyebrow HeightProportion (EtEHP);

FIG. 7D shows facial descriptor measurement Ear-to-Eye Height Proportion(EtEHP2);

FIG. 7E shows facial descriptor measurement Lobe Attachment Proportion(LAP);

FIG. 7F shows facial descriptor measurement Lobe Length Proportion(LLP);

FIG. 7G shows facial descriptor measurement Intertragical Notch DepthProportion (INDP);

FIG. 7H shows facial descriptor measurement Tragus ProtuberanceProportion (TPP);

FIG. 7I shows facial descriptor measurement Tragus Mid-Point Proportion(TMPP);

FIG. 7J shows facial descriptor measurement Antitragus ProtuberanceProportion (APP);

FIG. 7K shows facial descriptor measurement Antitragus Mid-PointProportion (AMPP);

FIG. 7L shows facial descriptor measurement Concha Inflexion Proportion(CIP);

FIG. 7M shows facial descriptor measurement Upper Concha SeparationAngle (UCSA);

FIG. 7N shows facial descriptor measurement Ear Tip Length Proportion(ETLP);

FIG. 7O shows facial descriptor measurement Upper Concha SeparationAngle (UCSA);

FIG. 7P shows facial descriptor measurement Ear Roundness Proportion(ERP);

FIG. 7Q shows facial descriptor measurement Crus Helix Angle (CHA);

FIG. 7R shows facial descriptor measurement Tubercle Proportion (TP);

FIG. 7S shows facial descriptor measurement Helix Thickness Proportion(HTP);

FIG. 7T shows facial descriptor measurement Lobule Inflexion Proportion(LIP);

FIG. 8 shows various relationships between genes, hormones, behavior,and facial features of an individual, such as a human or other animal;

FIG. 9 shows a system that can be used to implement any of the methodsof the present disclosure;

FIG. 10 shows an exemplary receiver operating characteristic (ROC)curve;

FIG. 11 shows a method for determining a predictor of a characteristicof an individual according to an embodiment of the present disclosure;

FIG. 12 shows a method for determining a characteristic of an individualaccording to another embodiment of the present disclosure;

FIG. 13 shows another method for determining one or more predictors of acharacteristic of an individual, according to other embodiments of thepresent disclosure;

FIG. 14 shows another method for determining a characteristic of anindividual according to other embodiments of the present disclosure;

FIG. 15 shows an example illustrating an embodiment of the method shownin FIG. 14;

FIG. 16 is a flowchart of a method for determining a predictor for thepairwise complementarity of two individuals for a particular purpose,according to other embodiments of the present disclosure;

FIG. 17 is a flowchart of a method for predicting a characteristic of apair of individuals, such as pairwise complementarity for a particularpurpose, according to other embodiments of the present disclosure;

FIG. 18 is a flowchart of an exemplary method for determining thesuitability of a new individual as a member of an existing group basedon the predicted conflict with the existing group members, according toother embodiments of the present disclosure; and

FIG. 19 shows a method for determining a strategy for a group ofindividuals using facial descriptor measurements, according to anotherembodiment of the present disclosure.

DETAILED DESCRIPTION

Various methods for using metrics, such as facial measurements,associated with the physical form of an individual, such as a human orother animal, to predict a characteristic including behavior,suitability, or potential are described. Also, the methods described areuseful for grouping, pairing, or matching of individuals for variouspurposes, both within a species and across species.

The biological mechanism relied upon in the disclosed embodiments wasfirst proposed in the ground-breaking 1999 Russian study entitled “EarlyCanid Domestication: The Fox Farm Experiment.” Using an extensivebreeding program of wild silver foxes (vulpes vulpes), this study showedthat selective breeding can be used to alter the innate personalitytraits or cognitive characteristics of a line of domesticated animals.More particularly, this study demonstrated that endocrine or hormonechanges are significantly correlated with both the personality traitchanges and a predictable suite of morphological changes, mostpredominantly in the structures of the face.

As illustrated graphically in FIG. 8, the innate personality of ananimal originates in its genetic composition. Genes dictate the basallevels of neurologically active hormones that control the behavior of ananimal, such serotonin, which inhibits aggression. Genes also dictatethe timing of the release of these hormones, such as corticosteroids(stress response) that control the windows of postnatal cognitive andsocial development in young animals. The cognitive framework of ananimal is thus determined from a combination of these innate personalitytraits provided by this genetically controlled endocrine makeup—theso-called “nature effect”—and the stimuli and experiences that theanimal was subject to during development—the so-called “nurture effect.”When viewed in the context of the animal's current environment, thiscognitive framework dictates its behavioral performance, which may bedefined in such terms as cognitive suitability to a specific task,success in performing a specific task, likelihood of displaying aspecific type or pattern of behavioral responses, or, when comparedagainst the personality types of its conspecifics, performance of anindividual in group situations.

As mentioned above, variation in the basal levels of neurologicallyactive hormones and in particular the timing of their release windowsduring development account not only for differences in innatepersonality among animals of the same species, but also for variabilityin morphology, particularly of the face. This direct correlation betweenthe facial structure and endocrine composition of an animal subsequentlyallows for quantifiable features of an animal's face to be correlatedwith and used as a proxy for predicting variability in the innatebehavior of individual animals as a function of their neurochemicalmakeup. Variations in facial structure may also be used to predict thebehavior and performance of an animal as a result of the variations inthe degree of functionality that they allow, in terms such as field ofvision, auditory acquisition, oxygen intake, feed intake, etc.

Various facial recognition and image matching techniques willmathematically model an animal's face and allow the prediction ofbehavior and performance. While these embodiments are effective, theprocesses and techniques of facial recognition and image matching aregenerally computationally intensive. Therefore, trigonometric modelingis used by some embodiments. Combinations of facial/shape recognition,image matching and trigonometric modeling may be used to predictbehavior and performance.

Since the domestication phenotype is common across many species ofanimals, embodiments described herein are applicable to a broad range ofindividuals, both mammals and non-mammals, including humans, donkeys,cattle, oxen, llamas, sheep, goats, turkey, geese, dogs, foxes, cats,ferrets, camels, geese, chickens, pigs, fish, etc. For example,embodiments described herein may be used to predict certaincharacteristics of humans via methods employing facial descriptormeasurements. During progressive stages of human embryonic growth,development of the brain and face remains intimately connected throughboth genetic signaling and biomechanical/biochemical mechanisms. Theface develops from populations of cells originating from the earlyneural crest, with cells from the neural tube gradually shifting to formthe prominences of the face. Differentiation of these early cells isclosely regulated through intricate genetic signaling mechanisms, withthe brain essentially serving as the platform on which the face grows.As these two structures continue to grow and develop during the laterembryonic stages, their phenotypes remain closely linked as complexgenetic hierarchies regulate patterns of cross talk between molecules,cells, and tissues.

For example, the close relationship between the functional developmentof the brain and structures of the face has been clearly documented fora number of developmental disorders. Among the most well known of thesedisorders is Fetal Alcohol Syndrome (FAS), which is the direct result ofexposure of the fetus to alcohol during pregnancy. FAS has been shown toresult in both an easily identifiable phenotype (i.e., collection ofminor facial abnormalities such as small eyes, smooth philtrum, thinupper lip) and developmental damage to the central nervous system thatis often permanent (e.g., speech delays, learning disabilities, poorreasoning skills, poor memory, attention deficit disorders, and low IQ).FIGS. 1A through 7T show a set of exemplary human facial descriptormeasurements that can be used to identify the phenotype associated withFAS. For example, FIGS. 1A through 1V show various facial descriptormeasurements related to the eye, while FIGS. 3A through 3S show variousfacial descriptor measurements related to the mouth.

By way of further example, Down Syndrome is another prenataldevelopmental disorder causing mental/social delays that yields aneasily identifiable phenotype including a host of distinctive facialfeatures such as small chin and/or mouth, round face, and roundedalmond-shaped eyes. Recent studies have even been able to identifymeasurable trends in facial features that distinguish between childrendiagnosed with Autism Spectrum Disorders (ASD) and those of typicaldevelopment. The facial descriptor measurements shown in FIGS. 1Athrough 7T may also be used to identify the phenotype associated withDown syndrome. However, as with FAS described above, this set of facialdescriptor measurements is merely exemplary, and may be reduced oraugmented for predicting Down syndrome or other human functionaldevelopment disorders within the spirit and scope of the presentdisclosure.

The set of human facial descriptor measurements shown in FIGS. 1Athrough 7T are merely exemplary, and the person of ordinary skill willrecognize that fewer than the entire set may be used to predict FAS orDown Syndrome. Furthermore, this set of human facial descriptormeasurements is not exhaustive and others may be incorporated. In someembodiments, two, three, or more of the facial descriptor measurementsare used in combination to predict a trait, characteristic, or syndromesuch as FAS or Down Syndrome. Moreover, while examples are shownprimarily with facial measurements, other head measurements and physicalmeasurements may be used alone, without facial measurements, or incombination with facial measurements. For example, measurements of thehead or crown may be used in conjunction with facial features to predictsyndromes or traits.

Given these established relationships between human facial structuresand cognitive development, any of the computationally inexpensive,two-dimensional, locally-normalized facial evaluation systems describedprovides a non-invasive analysis tool for use in multiple clinicalapplications. For example, embodiments of the facial analysis methodsand systems disclosed herein will diagnose children with specificcognitive development disorders based on the degree of divergencebetween their facial features and those of the overall typicalpopulation with respect to the phenotype for a disorder. Such adiagnosis tool is faster and less invasive than the standard cognitivetesting procedures currently employed, and may allow for earlierdiagnosis and interventions. More computationally expensive embodimentsor variations may also be used for diagnosis.

More generally, embodiments of the facial analysis methods and systemsdisclosed herein also can be used for predicting or inferring how anindividual is innately programmed to perceive, synthesize, and thenrespond to different types of stimuli. Facial analysis can be applied topredict not only how the cognitive framework of an individual willinteract with and be shaped by their environment, but also how it willinteract with the cognitive frameworks of other human beings, asdescribed in more detail below.

Each individual of a species (including humans) experiences the worlddifferently due to the unique way in which the individual perceives thesame set of stimuli. For example, pain is a stimulus whose perceptionvaries widely among individuals. Although cultural and socialexpectations arguably influence how readily an individual admits todiscomfort, it is clear that some people are more sensitive to pain thanothers, due in large part to greater capability to block out and copewith pain. Although some of this variability is due to the structure andnumber of neurons in the peripheral nervous system that register a paininput, how easily this input is brought to the forefront of a person'sconsciousness also plays a major role in overall pain sensitivity. Assuch, the cognitive structures for overcoming and dealing with suchunpleasant sensations play a major part in determining an individual'sresistance to pain. As a result, facial analysis methods can be appliedto develop a useful model or tool that objectively predicts anindividual's perception of the severity of pain.

Compared to the subjective pain assessments methods currently in use(e.g., scale of 1 to 10), such a model would benefit the medicalcommunity in many different ways. In addition simply helping doctorsmore efficiently and correctly assess the severity of an injury on theirinitial diagnosis, such a tool could also be used to more fairly assignpriority to patients in waiting room settings. An objective painassessment model also would be useful in preventative medicine andrecovery care, particularly in the case of patients with high painthresholds, who in underscoring the severity of their condition mightotherwise keep doctors from readily picking up on health concerns in itsearlier and more treatable stages. In addition, such a model could alsobe used in helping doctors to more rigorously qualify the strength andquantity of pain medications assigned to patients after a procedure. Forexample, if information from such an objective model indicates that apatient has a lower-than-average pain tolerance, a physician may decideto prescribe a higher strength pain medication to ensure that thepatient will be able to rest comfortably enough to recover fully fromtheir injury and to cope with the discomfort of their physical therapysufficiently enough to regain full mobility. Likewise, this objectivemodel might also indicate that a patient with a notably high painthreshold would do just fine with a lower level pain medicationfollowing a procedure that people with a normal pain threshold wouldotherwise find to simply have too difficult of a recovery period to copewith without the support of stronger medications. In this way, doctorscould tailor the amount of pain medication prescribed based on theindividual patient's objective pain perception, reducing the unnecessaryexposure of the patient and the overall distribution and subsequentunauthorized or illegal access to these highly addictive medications.

Another application of the facial analysis-based tool for objectivelypredicting an individual's pain perception is to help orthopedistsdetermine when surgery is an appropriate option. Major surgeries tojoints such as the knees, hips, and shoulders are typically followed bya recovery period during which most patients undergo a regular scheduleof painful physical therapy in order to regain full range of motion ofthe joint. Failure to follow this post-surgical therapy regime willresult in permanent stiffening and tightening of the joint, even to thepoint of complete immobilization—a so called “frozen joint.” Due to suchrisks, most orthopedists try to avoid major surgeries as long aspossible, typically until and even after chronic pain has begun toseriously impact the patient's quality of life. By using the informationgleaned from facial inference technologies about their patients' innatetolerance of or sensitivity to pain, orthopedists can more accuratelypredict how well a patient will be able to follow the post-surgicaltherapy and, thus, the likelihood of a full recovery. With this ability,orthopedists can make informed decisions tailored to the individualpatient about if and when the benefits outweigh the risk to pursuesurgical options.

Another group of applications for the facial analysis methods describedherein relate to predicting an individual's sensitivity to social cues.Individual humans display a wide range of sensitivity in their abilityto readily pick up on and correctly perceive social cues. On the mostsensitive side of this spectrum are Highly Sensitive Persons (HSPs), whoinnately pick up on even the subtlest of social cues. Such individuals,who are thought to make up roughly 15% of the population, are acutelyand at times even painfully aware of the thoughts and feelings of othersduring social situations, and are subsequently extremely sensitive tothe reactions of others. While they can be extremely responsive topraise, they also tend to take even the lightest criticisms andreprimands harshly. As a result, they are more heavily influenced andreflective of social experiences than most people, particularly duringtheir early childhood, and are more susceptible to being shy, becomingintroverted, and performing poorly at tasks when they feel that they areunder scrutiny.

On the opposite end of this spectrum are individuals with AutismSpectrum Disorders (ASDs), who show a very low sensitivity to perceivingthe social cues of others. In some ASDs, such as Asperger's Syndrome,individuals are unable to properly synthesize and correlate differentfacial expressions and tones of voice to the correct emotional state ofothers. In all ASDs, however, it is readily observed and understood thatsuch individuals simply do not pick up on social cues as readily as thegeneral population, typically because they are too distracted byextraneous environmental details to notice cues from their socialenvironment. As a result, attempting to communicate with individualssuffering from ASDs can be fairly frustrating for others who areunaccustomed to or lack the patience to deal with this lack of focus andresponsiveness. Individuals with ASDs, including those who are evendeemed high functioning, also tend to have a difficult time maintaininginterpersonal relationships due to their inability to connect and relatewith what others are feeling.

Distinct patterns in the facial features that distinguish individualswith ASDs from those in the general population have long been noted bythe medical and psychiatric communities. Moreover, recent research hasshown that computer-generated facial measures can identify anddifferentiate between children with such types of conditions. Using thefacial analysis methods described herein, similar models can be used toidentify HSPs and those suffering from ASDs from among the generalpopulation. In addition to identifying and classifying persons in theseoutlying groups, these facial analysis methods can be used to model thesocial sensitivity of the general population on a continuous scale, andto predict the social sensitivity of any one individual.

By further example, the facial analysis methods described herein alsocan be applied to model and predict how individuals respond toaggression. In general, the range of innate responses to aggressionvaries widely within the general population. Some individuals are bynature very easily offended, and tend to take the statements and actionsof others more personally than intended. Such individuals subsequentlytend to be much more bellicose in nature, and more readily engage inconfrontation with others. On the other hand, some individuals regardvery few statements and actions of others as offensive, even when theywere meant to be. These individuals are frequently seen as having a“long fuse” or low temper. The ability to objectively model and predictan individual's relative perception of aggressive behavior of others hasmany benefits. For example, such information would help determine thesuitability of an individual to certain tasks or jobs, particularlythose that require considerable interaction with customers, working in aculturally diverse team environment, or dealing with situations whereaggressive behavior is the norm (e.g., law enforcement). Also, such amodel could also prove valuable in dealing with criminals themselves,helping law enforcers to identify individuals that pose a greater riskof getting into physical confrontations either within the general publicor within a prison setting.

For example, a model based on a plurality of facial measurements may beused in the manner described below with reference to FIGS. 11 through 14to predict a particular individual's innate personality such asaggression and competitiveness. One or more of the facial descriptormeasurements shown in FIGS. 1A through 7T may also be used for thispurpose. However, this set of facial descriptor measurements is merelyexemplary and may be reduced or augmented as necessary within the spiritand scope of the present disclosure. Furthermore, multivariate analysisof a plurality of facial measurements statistically normalized againstlike measurements of a standard population of individuals can be used topredict other variable aspects of innate personality such as aggression,dominance, conscientiousness, compassion, extroversion, IQ, etc. Insightinto such personality traits of an individual could be used to predictvarious aspects of behavior and performance such as learning styles,athletic performance, business decisions, etc.

For example, each year Major League Baseball teams attempt to find thebest young players to select in the amateur draft. Although the draftcontains multiple rounds, the first round is especially importantbecause this is where the best players are chosen. Each of thesefirst-round draft picks are typically compensated by a multi-milliondollar signing bonus. Given their relative rarity, outstandingleft-handed pitchers often are the most highly sought-after first-rounddraft picks, and often command larger signing bonuses that otherplayers. However, many of these pitchers are relatively young andunproven. The cognitive suitability of a pitcher to performing under theextreme pressure situations found in Major League Baseball is a criticalcomponent for success that is otherwise very difficult to measure usingconventional tools currently available to baseball scouts. Althoughbaseball scouts use many different techniques (e.g., radar guns) tomeasure pitchers' physical capabilities, often these players' mental andcognitive capabilities—so-called “intangibles”—remain relatively unknownat draft time. Accordingly, it would be advantageous for teams to have areliable indicator of the mental and cognitive capabilities ofprospective first-round draft picks, such as left-handed pitchers, priorto making a decision to invest a large signing bonus and take theopportunity cost of foregoing another top prospect.

Facial and other physical measurements of successful left-handedbaseball pitching prospects (or any other baseball prospects) can bemade and compared with measurements of a standard population. Digitalimages and reference points may be used to ease the burden of themeasurement process. Ratios, angles, or nominal measurements may be usedinstead of or in addition to actual measurements. A number of metricsmay then be analyzed to find those metrics which show the greatestcorrelation or significance in determining statistically whether aperson is more likely than another to be a successful left-handedpitcher. Once the more significant metrics are identified, the systemmay simply rely on those metrics to predict a person's likelihood tosucceed as a left-handed pitcher. A baseball scout may use the system asa tool to assist in choosing one left-handed pitching prospect overanother by scoring both candidates. Although the advantages have beenillustrated by the example of a left-handed pitcher, those skilled inthe art will recognize that such advantages could be achieved by use ofthe same or similar methods with respect to other baseball players orplayers in other sports using a draft, such as the National FootballLeague (NFL) and the National Basketball Association (NBA). Moreover,those skilled in the art will recognize numerous other applications ofthe methods and techniques described throughout the specification tohumans.

FIG. 11 shows an embodiment in the form of a method for determining apredictor of a characteristic of a particular type of individual (e.g.,a human). As used herein, “type” refers to breed, species, sub-species,or any other relevant genetic similarity. Also, “characteristic” is usedbroadly to refer to a wide variety of features, traits, behaviors,capabilities, suitability, etc., including the most basic or fundamentalpersonality features (e.g., sensitivity); more complex personalityfeatures that may be comprised of multiple basic features; and overallsuitability for a particular task, purpose, relationship, etc., whichmay be influenced by a variety of fundamental and complex personalityfeatures. Accordingly, given the appropriate inputs, the method of FIG.11 can be used to predict any type or complexity of characteristic.Although FIG. 11 illustrates the one or more embodiments by blocksarranged in a specific order, this order is merely exemplary and thesteps or operations comprising the method may be performed in adifferent order than shown in the figure. Moreover, a person of ordinaryskill will understand that the blocks shown in FIG. 11 may be combinedand/or divided into blocks having different functionality.

In block 1100, a sample library is created. This step comprisesobtaining one or more digital images of an anatomical region for each ofa plurality of individuals, e.g., the face of a horse or a human. Theimage may be obtained in various ways, such as from a memory card of adigital camera, by downloading via File Transfer Protocol (FTP), viaemail, etc. Once obtained, the images are stored in a memory operablyconnected to a digital computer, such as memory that is either locallyor remotely accessible by the computer, including an attached harddrive, removable flash drive, network drive, RAID (redundant array ofindependent disks) drive, removable memory card, etc.

Block 1100 also comprises obtaining additional data related to thecharacteristic of interest and storing it in the same or a differentmemory operably connected to the digital computer. As used herein,“data” may refer to performance records, vital statistics, results ofbehavior studies, and/or any other quantitative information that isrelated in some way to the characteristic of interest. While qualitativedata may also be used, in many embodiments the qualitative data isconverted into quantitative data for easier use by the system. Theadditional data may pertain to individuals in the sample library, groupsof individuals in the sample library, or more generally the type ofindividual. Additional data may comprise information on breeding (e.g.,pedigree/ancestry), health, and environmental factors (e.g., training,experiences). The additional data is stored in a manner and location bywhich it can be associated with the sample library images, such as in arelational database accessible by the digital computer.

In block 1102, a plurality of reference points are added to the one ormore stored images of a particular individual in the sample library.This may be accomplished in an automated fashion or manually, forexample by use of a program with a graphical user interface (GUI)running on the digital computer. For example, the one or more images canbe processed by MATLAB, an advanced mathematical analysis program soldby The Mathworks, Inc., of Natick, Mass. (http://www.mathworks.com).MATLAB provides advanced image processing features and flexible optionsfor definition of large numbers of variables, specifically matrices.Reference points are added to each of the images by using the MATLAB“GInput” command, which provides an interactive selection GUI. In someembodiments, reference points are manually selected on the image. Onesuch embodiment is shown in FIGS. 1A through 7T, which were manuallyannotated with reference points (e.g., the four points in FIG. 1A). Inother embodiments, reference points may be added automatically by MATLABor another software application based on a generic model of theindividual's face. Once the reference points are entered onto an image,MATLAB maps their pixel locations within the image to numericalcoordinates within the corresponding matrix to facilitate furthercomputation.

In block 1104, one or more facial descriptor measurements (FDMs) relatedto the characteristic of interest are computed from the set of referencepoints that were added to the one or more digital images of theindividual. The facial descriptor measurements may be computed usingdistance measurements and trigonometric functions as illustrated above.Because length distance measurements are based on coordinate positionswithin the pixel matrix, the absolute distance values may be sensitiveto factors such as camera resolution, artifacts of one or morecompressions of the image, and cropping applied to isolate the face. Insome embodiments, the length measurements may be normalized tostructural reference lengths that are effectively constant amongindividuals of the same type and subject to the same set of factors. Inother embodiments, the length measurements may be normalized againstmeasurements (e.g., length or area) of other features in the same areaor region of the face. For example, a measure of the lip thickness maybe normalized by the lip length or width. However, it is apparent to oneof ordinary skill that the facial descriptor measurements may be basedupon absolute or non-normalized length measurements if the factorsdescribed above were not present or were not a concern. In otherembodiments, one or more of the facial descriptor measurements may bebased on an angular measurement or an area measurement. The facialdescriptor measurements may be based on non-trigonometric calculations,such integral calculus calculations, or a combination of trigonometricand non-trigonometric calculations.

In some embodiments, one or more of the digital images arethree-dimensional images. Such images may be created by combiningmultiple two-dimensional images using stereophotogrammetry or othermethods known to persons of ordinary skill in the art. In suchembodiments, one or more of the facial descriptor measurements may bebased on a three-dimensional measurement, such as an absolute distancemeasure, absolute volume, a volumetric ratio, a solid angle, a dihedralangle, a surface area, or a combination thereof.

As shown in FIG. 11, blocks 1102 and 1104 are repeated for eachindividual in the sample library. Once complete, in block 1108, arelationship is determined between a particular facial descriptormeasurement and the additional data related to the characteristic ofinterest using all individuals in the sample library. For example, therelationship can be determined from the mathematical correlation betweenthe facial descriptor measurement and additional data for allindividuals in the sample library. The correlation may be normalized orscaled as necessary to make it meaningful for further computation orinterpretation.

Other measures can be used to determine the relationship of a facialdescriptor measurement to the additional data. For example, incategorical models (i.e., those used to separate individuals intocategories and/or subcategories), receiver operating curve (ROC)analysis may be used to determine how effectively a facial descriptormeasurement categorizes the collection of individuals in the samplelibrary according to the characteristic of interest. In other words, ifthe additional information associated with the images in the samplelibrary includes the actual categories and/or subcategories that theindividuals fall into (e.g., personality type/subtype,consumption/purchasing patterns, learning style, type of event, etc.),ROC analysis can be used to determine how well a facial descriptormeasurement assigns individuals to their actual category (i.e., truepositive rate) while avoiding assignment of individuals to incorrectcategories (i.e., false positive rate). FIG. 10 shows an exemplaryreceiver operating curve, where the false positive rate is shown on thehorizontal axis and the true positive rate is shown on the verticalaxis. The curve shown in FIG. 10 has an area of 0.801; an area greaterthan 0.65 indicates that a particular facial descriptor measurement isuseful for prediction or categorization. Alternately, multi-dimensional,Euclidean distance analysis also may be used to separate two groupscategorically. Other methods for determining a relationship based onappropriate statistical models are apparent to persons of ordinary skillin the art. As illustrated in FIG. 11, block 1106 is repeated for eachfacial descriptor measurement.

In block 1112, one or more of the facial descriptor measurements areselected to be used as predictors of the characteristic of interest. Anynumber of facial descriptor measurements—up to and including the entireset—may be selected. If there are multiple characteristics of interest,then a separate selection may be made for each characteristic. Theselection may be made in various ways, depending on the availableinformation and the characteristic of interest. The selection may bebased on the raw facial descriptor measurements themselves, or uponscaled or transformed versions. For example, the z-score of a particularfacial descriptor measurement may be used for purposes of selection. Asknown to persons of ordinary skill, a z-score represents the relativefrequency (or rarity, as the case may be) of a particular measurementwithin a population, and is also commonly referred to as a standardscore or z-value. Other transformations will be apparent to persons ofordinary skill in the art.

Moreover, in block 1112, the combination of the selected facialdescriptor measurements that optimizes the predictor is also determined.In some embodiments, an optimal linear combination of the selectedsubset of facial descriptor measurements is determined using statisticalanalysis techniques. For example, in categorical models, ROC analysismay be used to select which combination of facial descriptormeasurements best categorizes the collection of individuals in thesample library according to the characteristic of interest. As describedabove with respect to block 1108, ROC analysis can be used to determinewhich combination of facial descriptor measurements best assignsindividuals to their actual category (i.e., true positive rate) whileavoiding assignment of individuals to incorrect categories (i.e., falsepositive rate). In the spirit of the disclosure, however, a non-linearcombination of the entire set, or a selected subset, of the facialdescriptor measurements also may be determined from the sample library.A non-linear combination of the selected facial descriptor measurementsmay be selected using optimization techniques including, but not limitedto, Newton's Method and Lagrange's Method. Moreover, the selected linearand/or non-linear combination may be of the facial descriptormeasurements themselves, or of scaled or transformed versions such asthe z-scores. If multiple characteristics are of interest, then anoptimal combination for each characteristic may be selected.

FIG. 13 is a flowchart of an alternative embodiment of a method fordetermining a predictor of a characteristic of a particular type ofindividual (e.g., a human). Although FIG. 13 illustrates the one or moreembodiments by blocks arranged in a specific order, this order is merelyexemplary and the steps or operations comprising the method may beperformed in a different order than shown in the figure. Moreover, aperson of ordinary skill will understand that the blocks shown in FIG.13 may be combined and/or divided into blocks having differentfunctionality.

The method of FIG. 13 explicitly recognizes that more complexcharacteristics are based on combinations of more fundamentalcharacteristics and overall suitability for a particular task, purpose,relationship, etc., may be based on a variety of fundamental and complexcharacteristics. Accordingly, the method of FIG. 13 can determine amodel comprised of predictors of basic characteristics (called“principal indicators”), which are based on facial descriptormeasurements; predictors of more complex characteristics (called“complex indicators”), which may be based on, or combinations of, facialdescriptor measurements and/or principal indicators; and predictors ofan ultimate suitability or category for an individual (called“suitability indicators”), which may be based on any of the facialdescriptor measurements, principal indicators, and complex indicators.Although not shown in FIG. 13, persons of ordinary skill will readilycomprehend that the model may comprise one or more additional levels ofindicators. For example, the model may comprise one or more “tertiaryindicators” that may be based on, or combinations of, facial descriptormeasurements, principal indicators, and/or complex indicators. In suchcase, the one or more suitability indicators may be based on any of thefacial descriptor measurements, principal indicators, complexindicators, and tertiary indicators. Moreover, models comprising suchadditional levels of indicators may be used in other methods describedherein for predicting characteristics of one or more individuals, suchas shown in and described below with reference to FIG. 14.

In block 1300, a sample library comprised of data for N individuals orsubjects is created. This operation comprises obtaining one or moredigital images of an anatomical region for each of the subjects, e.g., ahuman face. As discussed above with reference to FIG. 11, the images maybe two- or three-dimensional images. The images may be obtained invarious ways, such as from a removable or non-removable memory of acamera device (e.g., digital camera, webcam, smartphone, tablet, etc.),by downloading via File Transfer Protocol (FTP); by streaming from acamera device, via email, etc. Once obtained, the images are stored in amemory operably connected to a digital computer, such as memory that iseither locally or remotely accessible by the computer, including anattached hard drive, removable flash drive, network drive, RAID(redundant array of independent disks) drive, removable memory card,etc.

Block 1300 also comprises obtaining additional data related to thesubjects and storing it in the same or a different memory operablyconnected to the digital computer. The additional data may relatespecifically to the individuals in the sample library, groups ofindividuals in the sample library, or more generally to a population ofwhich the individuals are a part. As used herein, “additional data” mayrefer to performance records, vital statistics, results of behaviorstudies, environment data, and/or any other quantitative informationthat is related in some way to the characteristic of interest. Whilequalitative data may also be used, in many embodiments the qualitativedata is converted into quantitative data for easier use by the system.Moreover, additional data may be comprised of principal additional data,which are related to the principal indicators of interest; complexadditional data, which are related to the complex indicators ofinterest; and categorical additional data, which are related to theultimate suitability indicator or categorization. The additional data isstored in a manner and location by which it can be associated with thesample library images, such as in a relational database accessible bythe digital computer.

In block 1302, the images corresponding to each of the N individuals orsubjects in the sample library are annotated with a plurality ofreference points. As discussed above with reference to FIG. 11, theannotation may be done manually or automatically. Also in block 1302, aset of M facial descriptor measurements (Fi, i=1 to M) are computedbased on the reference points annotated onto the one or more images foreach of the N individuals. The facial descriptor measurements Fi may becomputed using absolute or normalized distance measurements, angularmeasurements, area measurements, volume measurements, etc. usingtrigonometric functions, integral calculus, and other computationalmethods known to persons of ordinary skill. Although block 1302 showsboth the annotation and the computation of Fi being done sequentiallyfor each subject, the person of ordinary skill will recognize that theseoperations could be arranged differently, e.g., annotations for allsubjects followed by computation of Fi for all subjects.

In block 1304, one or more relationships is determined between each ofthe facial descriptor measurements Fi, i=1 to M, and the principaladditional data across all N subjects in the sample library. By way ofexample, the one or more relationships for facial descriptormeasurements Fi can be determined using one or more mathematicalcorrelations between Fi and at least some portion of the principaladditional data for all N subjects in the library. The correlation maybe normalized, scaled, or transformed (e.g., z-score) as necessary tomake it meaningful for further computation or interpretation. Asdiscussed above with reference to FIGS. 11 and 12, in some embodiments,receiver operating curve (ROC) analysis may be used to determine theeffectiveness of facial descriptor measurement Fi as a principalindicator, based on the ability of Fi (or a function thereof) to assignindividuals to their actual category (i.e., true positive rate) whileavoiding assignment of individuals to incorrect categories (i.e., falsepositive rate). In other embodiments, correlation analysis may be usedto determine the one or more relationships. Other methods fordetermining a relationship based on appropriate statistical models areapparent to persons of ordinary skill in the art.

In block 1306, it is determined whether principal indicators Pk, k=1 toK, are to be used in the computation of the suitability indicator, S. Inthe case that only facial descriptor measurements Fi, i=1 to M, are tobe used in computing S, the method proceeds to block 1318. Otherwise,the method proceeds to block 1308, where one or more of the facialdescriptor measurements, Fi, are selected to be used to compute each ofprincipal indicators Pk, k=1 to K. Any number of facial descriptormeasurements—up to and including the entire set—may be selected for eachprincipal indicator. The selection may be made in various ways,depending on the available information and the particular principalindicator, Pk. The selection may be based on the raw facial descriptormeasurements themselves, or upon scaled or transformed versions. Forexample, the z-score of a particular facial descriptor measurement maybe used for purposes of selection. Other transformations will beapparent to persons of ordinary skill in the art.

In block 1308, the combination of the selected facial descriptormeasurements that optimizes each of principal indicators Pk is alsodetermined. In some embodiments, an optimal linear combination of theselected subset of facial descriptor measurements is determined. Inother words, if Pk=α1Fk1+α2Fk2+ . . . αnFkn, then the facial descriptormeasurements Fki, i=1 to n, and the linear coefficients αi, i=1 to n,are determined in block 1308. The linear combination may be determinedusing statistical analysis techniques. For example, in categoricalmodels, ROC analysis may be used to select the combination of facialdescriptor measurements that provides a Pk that best categorizes thesubjects in the sample library, e.g., according to at least a portion ofthe principal additional data. In the spirit of the disclosure, however,a non-linear combination of the entire set, or a selected subset, of thefacial descriptor measurements also may be determined based on theprincipal additional data. A non-linear combination of the selectedfacial descriptor measurements may be selected using optimizationtechniques including, but not limited to, Newton's Method and Lagrange'sMethod. Moreover, the selected linear and/or non-linear combination maybe of the facial descriptor measurements themselves, or of scaled ortransformed versions such as the z-scores.

In block 1310, it is determined whether complex indicators Cj, j=1 to J,are to be used in the computation of the ultimate suitability indicator,S. If not, then the method proceeds to block 1318. Otherwise, the methodproceeds to block 1312, where one or more relationships are determinedbetween each of the principal indicators Pk, k=1 to K, and the complexadditional data across all N subjects in the sample library. By way ofexample, the one or more relationships for principal indicators Pk canbe determined using one or more mathematical correlations between Pk andthe complex additional data for all N subjects in the library. Thecorrelation may be normalized, scaled, or transformed (e.g., z-score) asnecessary to make it meaningful for further computation orinterpretation. As discussed above, in some embodiments, therelationships can be determined based on receiver operating curve (ROC)analysis. In other embodiments, correlation analysis may be used toselect the complex indicators used for computing the ultimatesuitability indicator, S. In block 1312, one or more relationships arealso determined between each of the facial descriptor measurements Fi,i=1 to M, and the complex additional data across all N subjects in thesample library. ROC analysis, correlations, or other methods known topersons of ordinary skill may be used to determine these relationships.

Next, in block 1314, one or more of the facial descriptor measurements,Fi, and principal indicators, Pk, are selected to be used to computeeach of complex indicators Cj, j=1 to J. Any number of facial descriptormeasurements and principal indicators—up to and including the entire setof both—may be selected for each principal indicator. Moreover, theselection may be limited only to one or more of the principalindicators, Pk, i.e., the facial descriptor measurements are not used.The selection may be made in various ways, depending on the availableinformation and the particular complex indicator, Cj. The selection maybe based on the raw values of Fi and Pk, or upon scaled or transformedversions. For example, the z-score of a particular Fi or Pk may be usedfor purposes of selection. Other transformations will be apparent topersons of ordinary skill in the art.

In block 1314, the combination of the selected facial descriptormeasurements, Fi, and/or principal indicators, Pk, that optimizes eachof complex indicators Cj is also determined. In some embodiments, anoptimal linear combination of the selected group of Fi and/or Pk isdetermined. In other words, if Cj=α1Fk1+α2Fk2+αnFkn+β1Pk1+β2Pk2+βmPkm,then the facial descriptor measurements Fki, i=1 to n, and principalindicators Pki, i=1 to m, are selected in block 1312 and the linearcoefficients α ki, i=1 to n, and β ki, i=1 to m, are determined in block1314. The linear combination may be determined using statisticalanalysis techniques. In some embodiments, ROC analysis may be used toselect the combination of facial descriptor measurements and/orprincipal indicators that provides a Cj that best categorizes thesubjects in the sample library, e.g., according to at least a portion ofthe complex additional data. In other embodiments, correlation analysismay be used to select the combination of measurements.

In the spirit of the disclosure, however, a non-linear combination ofthe selected group of facial descriptor measurements and/or principalindicators also may be determined based on the complex additional data.A non-linear combination of the selected group of facial descriptormeasurements and/or principal indicators may be determined usingoptimization techniques including, but not limited to, Newton's Methodand Lagrange's Method. Moreover, the selected linear and/or non-linearcombination may be of the facial descriptor measurements and/orprincipal indicators themselves, or of scaled or transformed versionssuch as the z-scores.

In block 1314, one or more relationships are also determined betweeneach of the complex indicators Cj and the categorical additional dataacross all N subjects in the sample library. By way of example, the oneor more relationships for complex indicators Cj can be determined usingone or more mathematical correlations between Cj and the categoricaladditional data for all N subjects in the library. The correlation maybe normalized, scaled, or transformed (e.g., z-score) as necessary tomake it meaningful for further computation or interpretation. Asdiscussed above, in some embodiments the relationships can be determinedbased on receiver operating curve (ROC) analysis while in otherembodiments correlation analysis may be used.

In block 1316, it is determined whether only the set of complexindicators Cj, j=1 to J, are to be considered in the model forsuitability indicator, S. If so, then the method proceeds to block 1320.Otherwise, the method proceeds to block 1318, where, one or morerelationships are determined between each of the facial descriptormeasurements, Fi, and the categorical additional data across all Nsubjects in the sample library. In addition, one or more relationshipsalso are determined between each of the principal indicators, Pk, andthe categorical additional data across all N subjects in the samplelibrary. These relationships can be determined in the manner describedabove with respect to the relationships with the principal and complexadditional data.

In block 1320, one or more of the facial descriptor measurements, Fi,principal indicators, Pk, and complex indicators, Cj, are selected to beused to compute the suitability indicator, S. Any number of facialdescriptor measurements, principal indicators, and complex indicators—upto and including the entire set of all three—may be selected for eachsuitability indicator. Moreover, the selection may be limited only toone or more of the complex indicators, Cj, i.e., the facial descriptormeasurements and principal indicators are not used. The selection may bemade in various ways, depending on the available information and theparticular suitability indicator, S. The selection may be based on theraw values of Fi, Pk, and/or Cj, or upon scaled or transformed versions.For example, the z-score of a particular Fi, Pk, or Cj, may be used forpurposes of selection. Other transformations will be apparent to personsof ordinary skill in the art.

In block 1320, the combination of the selected facial descriptormeasurements, principal indicators, and complex indicators thatoptimizes suitability indicator S is also determined. In someembodiments, an optimal linear combination of the selected group of Fi,Pk, and/or Cj is determined. In other words, if S=α1Fk1+α2Fk2+ . . .αnFkn+α1Pk1+β2Pk2+ . . . βmPkm+γ1Ck1+γ2Ck2+ . . . γ rCkr then the facialdescriptor measurements Fki, i=1 to n, principal indicators Pki, i=1 tom, and complex indicators, Cki i=1 to r, are selected in block 1320 andthe linear coefficients αi, i=1 to n, βi, i=1 to m, and γi, i=1 to r aredetermined in block 1320. To the extent that the selection is limited toone or more of the complex indicators, Cj, the correspondingcoefficients αi and βi are zero. The linear combination may bedetermined using statistical analysis techniques. For example, in someembodiments, ROC analysis may be used to select the combination offacial descriptor measurements and/or principal indicators that providesa value for S that best categorizes the subjects in the sample library,e.g., according to at least a portion of the categorical additionaldata. In other embodiments, correlation analysis may be used to selectthe combination of measurements and/or indicators. In the spirit of thedisclosure, however, a non-linear combination of the selected group offacial descriptor measurements and/or principal indicators also may bedetermined based on the complex additional data, in the same or similarmanner as discussed above.

As mentioned above, the additional data stored in the sample library inblock 1100 of FIG. 11 and block 1300 of FIG. 13 may compriseenvironmental data. An individual's actual personality and behavior isinfluenced not only by their innate personality traits but also by theirenvironment, including experiences and social interactions throughoutlife, but particularly during early childhood. Accordingly, predictionof an individual's characteristic such as behavior or personality traitsshould take into account the relevant environmental factors to thegreatest extent possible. For example, the additional data stored in thesample library should include environmental information related towhether, or to what degree, each of the individuals has experiencedparticular events (e.g., death of a parent before a certain age), beenexposed to certain environmental stimuli (e.g., violence), or has aparticular family status (e.g., only child). When determining a modelfor predicting a characteristic of an individual, such as by using oneor more of the methods shown in FIGS. 11 and 13, environmental variablesmay be incorporated in several different ways. First, environmentalinformation about whether or not an individual has experienced aparticular event can be represented by a Boolean-valued environmentalvariable. If this event is associated with one or more facial descriptormeasurements, principal indicators, or complex indicators, theenvironmental variable can be used to enable and disable thecontributions of the associated measurements and/or indicators to theoverall predictor model. For example, if environmental variable τ isassociated with principal indicator P1, then complex indicator C thatdepends on P1 would be expressed as C=τ·α1·P1+α2P2+ . . . αnPn. If theenvironmental information relates to degree or type of experience, thecorresponding environmental variable (e.g., τ) can take on either adiscrete or a continuous range of values.

In other embodiments, environmental variables may be included asstandalone terms in the models determined for principal, complex, and/orcategorical indicators. For example, a complex indicator, C, thatdepends on environmental variable r as well as principal indicators P1through Pn could be expressed as C=α1·P1+α2·P2+ . . . αn·Pn+αn+1·τ. Thevalue for the linear coefficient αn+1 would then be determined togetherwith linear coefficients α1 through an, e.g., in block 1314 of FIG. 13.More generally, in such embodiments, environmental variables may betreated in the same manner as facial descriptor measurements, principalindicators, and complex indicators when determining the optimalselection and combination comprising a higher-level indicator.

According to the embodiment illustrated by FIG. 12, a subset andcombination of facial descriptor measurements selected using theembodiment illustrated by FIG. 11 then can be used to predict acharacteristic of an individual based on the facial descriptormeasurements for that individual. In other words, the subset andcombination selected based on the sample library can be applied to otherindividuals of the same type to determine the characteristic for thoseindividuals. Although FIG. 12 illustrates the one or more embodiments byblocks arranged in a specific order, this order is merely exemplary andthe steps or operations comprising the method may be performed in adifferent order than shown in the figure. Moreover, a person of ordinaryskill will understand that the blocks shown in FIG. 12 may be combinedand/or divided into blocks having different functionality.

In block 1200, digital images and additional data are obtained andstored for an individual of interest, in the same manner as describedabove with reference to the sample library (i.e., block 1100 of FIG.11). In block 1202, reference points consistent with those of the imagesin the sample library are added to the images of the individual ofinterest. In block 1204, facial descriptor measurements are calculatedfor the individual of interest.

In block 1206, the subset and combination selected in block 1112 isapplied to the facial descriptor measurements of the individual topredict the characteristic of interest. In block 1208, the samplelibrary optionally may be augmented by adding the image(s), additionaldata, and facial descriptor measurements for this individual.Subsequently, the method of FIG. 11 can be repeated using the augmentedsample library, and the resulting predictor can be applied to additionalindividuals of the same type in accordance with the method of FIG. 12.Although the embodiment illustrated by FIG. 12 has been described asbeing used in conjunction with the embodiment of FIG. 11, it also can beutilized in conjunction with the embodiment of FIG. 13. For example, themethod shown in FIG. 12 can be used to compute one of more of theprincipal indicators, Pk that are comprised of a linear or non-linearcombination of facial descriptor measurements, Fi.

FIG. 14 is a flowchart of an alternative embodiment of a method forpredicting a characteristic of an individual, such as suitability for aparticular task, purpose, relationship, etc., based on the facialdescriptor measurements for that individual. The embodiment illustratedby FIG. 14 may utilize the facial descriptor measurements together withthe models for principal indicators and complex indicators to determinea suitability indicator, S. The models for the principal and complexindicators may be determined by the method illustrated in FIG. 13 or byany other method that generates such models. Although FIG. 14illustrates the one or more embodiments by blocks arranged in a specificorder, this order is merely exemplary and the steps or operationscomprising the method may be performed in a different order than shownin the figure. Moreover, a person of ordinary skill will understand thatthe blocks shown in FIG. 14 may be combined and/or divided into blockshaving different functionality.

In block 1400, digital images and additional data are obtained andstored for a subject, in the same manner as described above withreference to the sample library (e.g., block 1100 of FIG. 11 and block1300 of FIG. 13). In block 1402, the images of the subject are annotatedwith a plurality of reference points. As discussed above with referenceto FIGS. 11 through 13, the annotation may be done manually orautomatically. In block 1404, a set of M facial descriptor measurements(Fi, i=1 to M) are computed based on the reference points annotated ontothe one or more images of the subject.

In block 1406, it is determined whether principal indicators are to beused in the computation of the suitability indicator, S. In the casethat only facial descriptor measurements Fi, i=1 to M, are to be used incomputing S, the method proceeds to block 1414. Otherwise, the methodproceeds to block 1408, where principal indicators Pk, k=1 to K, arecomputed using facial descriptor measurements Fi and the modeldetermined, for example, in block 1308 of FIG. 13. Subsequently, themethod proceeds to block 1410, where it is determined whether complexindicators are to be used to compute the suitability indicator. If not,then the method proceeds to block 1414. Otherwise, the method proceedsto block 1412 where complex indicators Cj, j=1 to J, are computed usingfacial descriptor measurements Fi and/or principal indicators Pk, basedon, the model determined, for example, in block 1314 of FIG. 13.

Next, in block 1414, the suitability indicator, S, for the subject ofinterest is computed based on the facial descriptor measurements Fi,principal indicators Pk, and/or complex indicators Cj using the modeldetermined, for example, in block 1320 of FIG. 13. In block 1416, thesubject is categorized based on the value of suitability indicator, S,and a suitability threshold, Sx. In some embodiments, thischaracterization may comprise determining that the subject is suitablefor a particular task, purpose, relationship, etc. based on comparingthe values of S and Sx, e.g., S>Sx or |S−Sx|<ε. In some embodiments,this characterization may comprise assigning the subject to one categoryif S>Sx and another category if S≤Sx. Other possible comparisons will beapparent to persons of ordinary skill in the art. Finally, in block1418, the sample library optionally may be augmented by adding theimage(s), additional data, facial descriptor measurements, principalindicators, complex indicators, and/or suitability indicator for thesubject.

FIG. 15 further illustrates the method of FIG. 14 by example. Initially,principal indicators P1, P2, and P3 are computed based on linearcombinations of facial descriptor measurements F1, F2, and F3. The useof the same facial descriptor measurements in computing each of theprincipal indicators is merely exemplary. Coefficients αi, βi, and γiused to compute the principal indicators may have been previouslydetermined, for example, by the method shown in FIG. 13. Z-scores Z1,Z2, and Z3, are determined for principal indicators P1, P2, and P3,respectively. Next, complex indicators C4 and C5 are computed by anon-linear combination of principal indicator z-scores Z1, Z2, and Z3.Coefficients αi, βi, γi, and φi used to compute the complex indicatorsmay have been previously determined, for example, by the method shown inFIG. 13. Z-scores Z4 and Z5 are determined for complex indicators C4 andC5, respectively.

Subsequently, categorical indicator for suitability for a task Stask iscomputed from a non-linear combination of z-scores Z1, Z2, and Z3corresponding to the principal indicators and z-scores Z4 and Z5corresponding to the complex indicators. Coefficients αi, βi, γi, and φiused to compute the suitability indicator Stask may have been previouslydetermined, for example, by the method shown in FIG. 13. Next, Stask maybe used to determine whether the subject with facial descriptormeasurements F1, F2, and F3 is suitable for the task at issue. This maybe done in a variety of ways, two of which are shown in FIG. 15. Forexample, the raw value of Stask may be compared against the threshold Sxto determine which of two categories to place the subject (e.g., “(+)”and “(−)” in FIG. 15). In another example, a z-score Ztask computed fromStask may be compared to a cutoff or threshold on the distribution ofthe population of interest (e.g., the 80th percentile) to determinewhether the subject is suitable for the task of interest. The populationof interest may be the general population or a specific sub-population,e.g., those who already are engaged in the particular task. The cutoffor threshold may be chosen in conjunction with the choice of population.As noted above, FIG. 15 is merely an example illustrating the embodimentshown in and described above in reference to FIG. 14; other variationswill be apparent to the person of ordinary skill.

In other embodiments, Stask may comprise a set of z-scores, Zq, q=1 toQ, each corresponding to one or more principal indicators Pk and/or oneor more complex indicators Cj, and threshold Sx may comprise a set ofthresholds, Sxq, q=1 to Q, each corresponding to a Zq. In suchembodiments, each of z-scores in {Zq} can be compared to itscorresponding threshold in {Sxq}, with subject categorized as suitablefor the task of interest if each member of {Zq} has the desiredrelationship—less or greater than, as the case may be—with thecorresponding member of {Sxq}. Alternately, the subject may bedetermined to be suitable if a subset of {Zq} and {Sxq} have the desiredrelationship (e.g., majority). In some embodiments, one or more of thethresholds {Sxq} may comprise a pair of thresholds related to thecorresponding member of {Zq}, in which case the desired relationship maybe between the pair of thresholds, e.g., Sxq1<Zq≤Sxq2. In someembodiments, Stask may comprise a set of raw values, {Sq}, eachcorresponding to one or more principal indicators Pk, and/or one or morecomplex indicators Cj, and threshold Sx may comprise a set of thresholds{Sxq}, each corresponding to a member of {Sq}. Suitability may bedetermined for these embodiments in the same manner as discussed abovewith respect to embodiments comprising z-scores.

In the embodiments described above, the operation in block 1416 of FIG.14 categorizes the subject as suitable or non-suitable based on thesuitability indicator. In other embodiments, however, the operation inblock 1416 may comprise using the suitability indicator to assign thesubject to one of a plurality, or set, of categories. This set ofcategories may be exhaustive (i.e., every member of a population will beplaced in one of the categories) or non-exhaustive, in which case theset may comprise an “other” or similarly-named category to whichsubjects that do not fall into one of the defined categories areassigned. In some embodiments, a set of categories Kt, t=1 to T, aredefined by a set of thresholds, Sxt, t=1 to T+1, such that a subject isassigned to category Kt if Sxt<Stask≤Sx(t+1). Optionally, beginningthreshold Sx1 and end threshold Sx(T+1) may be set to −/+∞,respectively.

In other embodiments, Stask may comprise a set of z-scores, Zq, q=1 toQ—or raw scores, Sq, q=1 to Q—each corresponding to one or moreprincipal indicators Pk and/or one or more complex indicators Cj. Eachcategory Kt is defined by a pair of values {Sxqtmin, Sxqtmax} defining arange for each z-score of the set {Zq}, i.e., a set of Q ranges. Asubject will be assigned to a category Kt if each member of the set ofz-scores {Zq} falls within the range {Sxqtmin, Sxqtmax} associated withthat z-score and category. In other words, a subject will be assigned toa particular category only if all z-scores (or raw scores, as the casemay be) fall within the set of ranges associated with that category. Inother embodiments, a subject may be assigned to a particular category ifa required subset of the z-scores (or raw scores) falls within the setof ranges associated with that category. The required subset may bedetermined in various ways, such as by majority (i.e., at least Q/2) ofthe set, by majority of each of two subsets, by all of a one subset andone or more of the remainder, etc. In these embodiments, the set ofcategories {Kt} may be exhaustive (i.e., every individual in thepopulation will be assigned to one of the categories) or non-exhaustive.

For example, embodiments of the present disclosure may be used topredict the risk level of a driver for purposes of automobile insurance.Accordingly, Stask may comprise a set of z-scores, Zq, q=1 to Q—or rawscores, Sq, q=1 to Q—each related to a particular risk area for a driver(e.g., specific types of accidents, theft risk, etc.). In someembodiments, the value of a particular z-score Zq is proportional to therisk level in the corresponding risk area. An insured driver (orcandidate for insurance) could be assigned to one of a plurality ofoverall risk categories {Kt, t=1 to T} based on the values of each ofthe set of z-scores {Zq} relative to one or more correspondingthresholds. In such embodiments, additional data used to determine themodel for categorization of drivers may comprise frequency and/orseverity of past accidents, overall cost of accidents to insurancecompany, number of traffic violations, traffic density in individual'shome city, length of commute (if any), etc. In some embodiments, such amodel may be applied using additional data and facial descriptormeasurements pertaining to a new individual to predict that individual'srisk level.

In other embodiments, the set of categories may be related to apre-existing population categorization model, such as a pre-existingmodel used by insurance companies to categorize drivers and personalitycategorization models such as the Myers Briggs Type Indicator (MBTI),the Kiersey Temperament Sorter, the Costa McRae Five Factor model (alsoknown as the Big Five model), the Petrides Trait EI model for emotionalintelligence, and others known to persons of ordinary skill in the art.For example, the MBTI separates individuals into sixteen personalitycategories using subjects' answers to a psychometric questionnairedesigned to measure psychological preferences in how people perceive theworld and make decisions. The MBTI categories are based on the fourdichotomies of extraversion/introversion (E/I), sensing/intuition (S/N),thinking/feeling (T/F), and judging/perceiving (J/P).

The method shown in FIG. 13 could be employed to predict which of the 16categories that an individual would be assigned by an MBTI assessment.For example, the sample library created in block 1300 may comprisedigital images of individuals whose personalities have been categorizedusing MBTI, together with additional data for each individual comprisingtheir answers to the questionnaire, their rating on each of the fourdichotomies, and any other information used for MBTI categorization(e.g., gender). In blocks 1302 through 1318, a variety of facialdescriptor measurements, principal indicators, and complex indicatorscould be determined as described above. In some embodiments, a set offour predictors {SE/I, SS/N, ST/F, SJ/P} corresponding to the four MBTIdichotomies are determined in block 1320. The set of predictors may bebased on raw scores, z-scores, or a combination thereof. An indicator ofan individual's MBTI category can be determined based on this set ofdichotomy predictors.

These predictors may then be used according to the method shown in FIG.14 to predict to which of the sixteen personality types that anindividual subject would be assigned based on an MBTI assessment. Thesubject's predicted rating on the four MBTI dichotomies can bedetermined in block 1416 by comparing the subject's four dichotomyindicators {SE/I, SS/N, ST/F, SJ/P} to their corresponding thresholds{SxE/I, SxS/N, SxT/F, SxJ/P}. By way of example, if SE/I<SxE/I, then thesubject is likely to be categorized as an introvert (“I”); otherwise thesubject is likely to be categorized as an extrovert (“E”) by an MBTIassessment. The subject's ultimate MBTI category is predicted based onthe results of the four comparisons. In another embodiment, a singlecategory predictor SMBTI may be determined in block 1320 and used tocompute a single categorical indicator for a subject in block 1416. Anindicator of the subject's MBTI category may be determined based uponthe range of values that the categorical indicator falls within, asdescribed above.

While some embodiments of the present disclosure are useful forpredicting categorizations of individuals under existing models, such asthe MBTI, other embodiments can be used to augment existingcategorization models with additional, more granular information. Forexample, in addition to predicting a subject's rating on each of thefour MBTI dichotomies, the methods of FIGS. 11 through 15 can be used todetermine indicators of the strength of the subject's rating on each ofthe categories. The four dichotomy indicators {SE/I, SS/N, ST/F, SJ/P}may be expressed as z-scores such that a high positive value for adichotomy (e.g., E/I) indicates that the subject is strongly orientedtoward one side (e.g., E), a high negative value indicates that thesubject is strongly oriented toward the other side (e.g., I), and avalue near zero indicates that the subject is relatively balanced.Alternately, the four dichotomy indicators may be expressed aspercentile scores, e.g., with zero and 100 percentiles representing thetwo extremes of a dichotomy (e.g., extreme introverts and extremeextroverts). Continuously-valued indicators, such as these examples, maybe used to create sub-categories for one or more of the categories ofthe MBTI or other existent model.

Although exemplary embodiments have been discussed above in relation tohumans, embodiments of the methods for predicting a characteristic of anindividual may be applied for predicting various characteristics of anon-human animal. For example, embodiments may be employed to predictthe characteristic of gluttony in animals, i.e., how much a specificanimal will eat in relation to the population of other animals of thesame type. Moreover, such embodiments may be employed to predict how theanimal's food intake level is affected by environmental stress (e.g.,heat, handling, transport, weaning, parturition, injury, etc.) based onthe characteristics of their susceptibility to stress and their abilityto respond to it. Furthermore, embodiments may be employed to predict ananimal's preferences for particular types of food (based, e.g., ontexture, taste, smell, etc.) and how this contributes to variability inthe animal's food intake level. Embodiments also may be used to predicthow well an animal is able to digest their food.

Similarly, embodiments may be employed to predict how an animal willrespond to changes in their environment such as, for example, changes topen or stall, pen- or stallmates, transportation, feeding systems,feeding or other schedules, etc. Embodiments may be used to predict theanimal's susceptibility to stress, the expected level of stress theanimal would suffer in response to various stress-inducing changes, andtype and/or degree of physiological reaction to such changes includingdepression, aggression, flight, weight loss, diarrhea, self-injury,death, etc. Based on the predictions of such characteristics, suchembodiments could be used to select animals better suited to thrive interms of growth and/or production under the known stress of existinganimal management protocols.

Although the methods of FIGS. 11 through 15 have been described above interms of determining a predictor of a characteristic or indicator of anindividual (or predicting the characteristic or indicator using thepredictor, as the case may be), embodiments of the present disclosurealso may be used to determine the degree to which two individuals aresuitable for a particular purpose or type of relationship, i.e.,“pairwise complementary.” These embodiments may be used to determine iftwo individuals are pairwise complementary for any type of relationship,including husband-wife, student-teacher, athlete-coach/trainer,teammates, roommates, worker-supervisor, co-workers, business partners,etc.

For example, FIG. 16 shows a flowchart for a method for determining apredictor for the pairwise complementarity of two individuals for aparticular purpose, e.g., marriage. As will be apparent during thefollowing discussion, the embodiment illustrated by FIG. 16 is similarto the embodiment shown in and described above with reference to FIG.13. Although FIG. 16 illustrates the one or more embodiments by blocksarranged in a specific order, this order is merely exemplary and thesteps or operations comprising the method may be performed in adifferent order than shown in the figure. Moreover, a person of ordinaryskill will understand that the blocks shown in FIG. 16 may be combinedand/or divided into blocks having different functionality.

In block 1600, a sample library comprised of data for N individuals orsubjects is created. These N individuals are comprised of N/2 pairs ofindividuals that are associated based on an existing relationship of thetype for which a predictor will be determined. For instance, the Nindividuals may comprise N/2 married couples. The operation comprisesobtaining one or more digital images of an anatomical region for each ofthe individuals, as discussed above with reference to block 1100 of FIG.11 and block 1300 of FIG. 13 with block 1600 also storing additionaldata related to the individuals and their pairings in the samplelibrary. This additional data may comprise individual additional data,such as discussed above with reference to block 1100 of FIG. 11 andblock 1300 of FIG. 13, and pairwise additional data, which relates tothe pairing of the two individuals. The pairwise additional data maycomprise identifiers for the two individuals that are paired,quantitative information related to the duration, success, performance,etc. of the pairing, and environmental data related to the pairing ofthe individuals (e.g., existence and/or degree of certain externalevents or stimuli). The pairwise additional data may relate specificallyto individual pairs in the sample library, a plurality of pairs in thesample library, or more generally to a population of which the pairs area part. The additional data is stored in a manner and location by whichit can be associated with the sample library images, such as in arelational database accessible by a digital computer.

In block 1602, the images corresponding to each of the N individuals orsubjects in the sample library are annotated with a plurality ofreference points. Also in block 1602, a set of M facial descriptormeasurements (Fi, i=1 to M) are computed based on the reference pointsannotated onto the one or more images for each of the N individuals. Inblock 1604, one or more of the facial descriptor measurements, Fi, areselected to be used to compute each of individual principal indicatorsPk, k=1 to K. Also in block 1604, the combination of the selected facialdescriptor measurements that optimizes each of individual principalindicators Pk is also determined. In block 1606, one or more of thefacial descriptor measurements, Fi, and individual principal indicators,Pk, are selected to be used to compute each of individual complexindicators Cj, j=1 to J. Also in block 1606, the combination of theselected facial descriptor measurements, Fi, and/or individual principalindicators Pk that optimizes each of individual complex indicators Cj isalso determined. The operations in blocks 1602, 1604, and 1606 may besubstantially the same as the operations described above with referenceto blocks 1302 through 1314 of FIG. 13. Furthermore, the operation inblock 1606 may be omitted if no complex indicators are utilized in theprediction model being determined. In such embodiments, the methodproceeds from block 1604 to block 1608.

In block 1608, for each of the principal indicators Pk, one or morerelationships are determined between the pairwise additional data andone or more functions f(Pkx, Pky) of the individual principal indicatorPk for an associated pair of individuals (x, y), across all N/2associated pairs in the sample library. Also in block 1608, for each ofthe complex indicators Cj, one or more relationships are determinedbetween the pairwise additional data and one or more functions f(Cjx,Cjy) of the individual complex indicator Cj for the associated pair (x,y), across all N/2 associated pairs in the sample library. The functionsf(Pkx, Pky) and f(Cjx, Cjy) may be any linear or non-linear function ofthe pair of the particular individual indicator for both individuals inthe pair. Exemplary functions f (Pkx, Pky) include, but are not limitedto, Pkx, Pky, (α·Pkx+β·Pky), (α·Pkx−β·Pky), 1/|Pkx−Pky|, max(Pkx, Pky),(Pkx·Pky), (Pkx/Pky), etc., and like functions for f (Cjx, Cjy). Someembodiments also may include discontinuous functions. In someembodiments, the operations of block 1608 may comprise determining oneor more relationships between functions of multiple individual principalor complex indicators for pairs of individuals, e.g., f (Pk1 x, Pk1 y,Pk2 x, Pk2 y) for individual principal indicators Pk1 and Pk2.

In block 1608, for each of the principal and complex indicators, the setof pairwise functions f(Pkx, Pky) and f(Cjx, Cjy) to be analyzed for arelationship with the pairwise additional data may be determined invarious ways. In some embodiments, a fixed set is always analyzed, e.g.,Pkx, Pky, (α·Pkx+β·Pky), (α·Pkx−β·Pky), 1/|Pkx−Pky|, max(Pkx, Pky),(Pkx·Pky), and (Pkx/Pky), for relationships with the additional data.One or more of the functions in the fixed set may be selected for eachprincipal and/or complex indicator based on categorical potential withrespect to the additional data using, e.g., ROC or correlation analysisas described above. In other embodiments, one or more candidatefunctions of each principal and/or complex indicator may be selected byusing heuristic computational methods. For example, a larger list ofcandidate functions may be organized into a multi-stage search tree. Ateach stage, some small number of (e.g., two) candidate functions—ortypes of functions—are analyzed to determine which gives the bestcategorical potential based on the additional data. The best of thecandidates is selected and used in the next stage of the search tree. Inthis manner, a small set of candidate functions of each of theindividual principal and/or complex indicators can be identified fromamong a larger set of candidate functions. Such heuristic computationalalgorithms also may comprise artificial intelligence functions in whichresults of previous selections related to the same characteristic orindicator may be stored and used to guide future selections, as known bypersons of ordinary skill in the art.

The operation in block 1608 of selecting the pairwise functions f(Pkx,Pky) and f(Cjx, Cj y) may be broken into steps that are carried outsequentially. In some embodiments, the operation of block 1608 maycomprise first selecting one or more pairs of principal indicators (Pkx,Pky) and/or complex indicators (Cjx, Cjy) then determining the optimalfunctions f(Pkx, Pky) and f(Cjx, Cjy) of the selected pairs. In otherembodiments, the selection of the indicator pairs and the determinationof the optimal pairwise functions may comprise a substantially unitaryoperation. Other ways of structuring the operations of block 1608 willbe apparent to persons of ordinary skill in the art. Regardless of theselection process used, however, the candidate pairwise functionsanalyzed may comprise functions of individual principal indicators Pkxand/or Pkx, and functions of individual complex indicators Cjx and/orCjy. As used herein unless specifically stated to the contrary, the term“pairwise function” comprises functions of individual principal orcomplex indicators, or other such terms. Similarly, a function of twovariables f(x, y) comprises functions f(x) and f(y) of the individualvariables.

Moreover, although the operation of block 1608 is described above in thecontext of finding pairwise functions of the same type of indicator(i.e., principal or complex), this is merely exemplary and pairwisefunctions of different types of indicators may be selected in someembodiments. For example, embodiments of block 1608 may select one ormore functions f(Pkx, Pky, Cjx, Cjy) of both principal and complexindicators for the two individuals x and y. Furthermore, embodiments ofblock 1608 may select one or more functions of a combination ofprincipal indicators, complex indicators, and facial descriptormeasurements, Fi, corresponding to the pair of individuals x and y.Accordingly, although the description of operations of subsequent blocksof FIG. 16 may refer to pairwise functions f(Pkx, Pky) and/or f(Cjx, Cjy), persons of ordinary skill in the art will understand that theseoperations may encompass functions of various indicators andmeasurements pertaining to two individuals x and y.

In any event, the result of block 1608 is one or more pairwise functionscorresponding to the individual principal and/or complex indicators,together with their respective relationships to the pairwise additionaldata. In block 1610, some portion of the group of the pairwise functionsdetermined in block 1608 are selected to be used to compute the pairwisecomplementarity indicator, S. Any number of the pairwise functions·up toand including the entire set—may be selected. Moreover, the selectionmay be limited only to functions f(Cj x, Cj y) of individual complexindicators, i.e., functions f(Pkx, Pky) of individual principalindicators are not used. The selection may be made in various ways,depending on the available information and the particular pairwisecomplementarity indicator, S. The selection may be based on the rawvalues of functions f(Pkx, Pky) and/or f(Cj x, Cj y), or upon scaled ortransformed versions. For example, the z-score of a particular functionsf(Pkx, Pky) or f(Cjx, Cjy) may be used for purposes of selection. Othertransformations will be apparent to persons of ordinary skill.

The selection of the pairwise functions f(Pkx, Pky) and/or f(Cjx, Cjy)to be used for computing indicator S may be accomplished in variousways. In some embodiments, some fixed number of pairwise functionshaving the highest categorical potential with respect to the additionaldata may be selected. In other embodiments, one or more pairwisefunctions may be selected by using heuristic computational methods. Forexample, the various functions can be broken into groups or categoriesusing qualitative information related to the types of characteristicsthat they indicate. Subsequently, one or more pairwise functions may beselected from each group based on, e.g., their categorical potentialwithin the group. Alternately, selections may be constrained to a subsetof the groups based on the qualitative information related to thegroups. Such embodiments may comprise learning-based functionalitywhereby results of previous selections related to the same or similarindicator(s) may be stored and used to guide future selections. Othermethods for selecting pairwise functions f(Pkx, Pky) and/or f(Cjx, Cjy)to be used for computing indicator S will be apparent to the person ofordinary skill in the art.

In block 1610, a combination of the selected pairwise functions f(Pkx,Pky) and f(Cjx, Cjy) that optimizes pairwise complementarity indicator Sis also determined. In some embodiments, an optimal linear combinationof the selected pairwise functions f(Pkx, Pky) and f(Cj x, Cj y) isdetermined. In other words, if S=α1 f1(P1 x, P1 y)+α 2 f2(P2 x, P2 y)+ .. . α n fn(Pnx, Pny)+β1 f1(C1 x, C1 y)+β2 f2(C2 x, C2 y)+ . . . βmfm(Cmx, Cmy) then the pairwise functions f(Pix, Piy), i=1 to n, and f(Cix, Ci y), i=1 to m, are selected and the linear coefficients αi, i=1 ton, and βi, i=1 to m, are determined in block 1610. To the extent thatthe selection is limited to one or more of pairwise functions f(Cix, Ciy), the corresponding coefficients αi are zero. The linear combinationmay be determined using statistical analysis techniques. For example, incategorical models, ROC analysis may be used to select the combinationof pairwise functions f(Pkx, Pky) and/or f(Cjx, Cjy) that provides avalue for S that best predicts or categorizes the pairs in the samplelibrary, e.g., according to at least a portion of the pairwiseadditional data. In the spirit of the disclosure, however, a non-linearcombination of the selected group of pairwise functions f(Pkx, Pky)and/or f(Cjx, Cj y) also may be determined based on the pairwiseadditional data. A non-linear combination of the selected group ofpairwise functions f(Pkx, Pky) and/or f(Cjx, Cjy) may be determinedusing optimization techniques including, but not limited to, Newton'sMethod and Lagrange's Method. The selected linear and/or non-linearcombination may be of the pairwise functions f(Pkx, Pky) and/or f(Cjx,Cjy) themselves, or of scaled or transformed versions such as z-scores.

Although not shown in FIG. 16, persons of ordinary skill will readilycomprehend understand that the model may comprise one or more additionallevels of indicators. For example, the model may comprise one or more“tertiary indicators” that may be based on, or combinations of, facialdescriptor measurements, principal indicators, and/or complexindicators. In such case, the one or more suitability indicators may bebased on any of the facial descriptor measurements, principalindicators, complex indicators, and tertiary indicators. Moreover,models comprising such additional levels of indicators may be used inother methods described herein for predicting characteristics of one ormore individuals, such as shown in and described below with reference toFIGS. 17 and 18.

FIG. 17 is a flowchart of an embodiment of a method for predicting acharacteristic of a pair of individuals, such as pairwisecomplementarity for a particular purpose (e.g., marriage), based on thefacial descriptor measurements for those individuals. The embodimentillustrated by FIG. 17 may utilize models for principal indicators,complex indicators, and pairwise indicator functions to determine apairwise complementarity indicator, S. The models for principalindicators, complex indicators, and pairwise indicator functions may bedetermined by the method illustrated in FIG. 16 or by any other methodthat generates such models. Although FIG. 17 illustrates the one or moreembodiments by blocks arranged in a specific order, this order is merelyexemplary and the steps or operations comprising the method may beperformed in a different order than shown in the figure. Moreover, aperson of ordinary skill will understand that the blocks shown in FIG.17 may be combined and/or divided into blocks having differentfunctionality.

In block 1720, digital images and additional data are obtained andstored for a pair of individuals, in the same manner as described abovewith reference to the sample library (e.g., block 1600 of FIG. 16). Inblock 1722, the images of the subject are annotated with a plurality ofreference points. In block 1724, a set of facial descriptormeasurements, Fi, i=1 to M, are computed based on the reference pointsannotated onto the one or more images of the subject. In block 1726,individual principal indicators Pk, k=1 to K, are computed using facialdescriptor measurements Fi and the individual principal indicator modeldetermined, for example, in block 1704 of FIG. 16. In block 1728,individual complex indicators Cj, j=1 to J, are computed using facialdescriptor measurements Fi and/or individual principal indicators Pkbased on the model determined, for example, in block 1706 of FIG. 16. Inblock 1730, pairwise functions f(Pk1, Pk2) and/or f(Cj1, Cj2) ofprincipal and/or complex indicators for the two individuals (i.e.,individuals 1 and 2) are computed, and further used to compute pairwisecomplementarity indicator, S, of the prospective pair using the modeldetermined, for example, in block 1610 of FIG. 16.

Next, in block 1732, the subject is categorized based on the values ofthe pairwise complementarity indicator, S and a suitability threshold,Sx. In some embodiments, this characterization may comprise determiningthat the pair of individuals are suitable for a particular purpose orrelationship (e.g., marriage) based on comparing the values of S and Sx,e.g., S>Sx or |S−Sx|<ε. In some embodiments, this categorization maycomprise assigning the pair to one category if S>Sx and another categoryif S≤Sx. Other possible comparisons will be apparent to persons ofordinary skill in the art. Finally, in block 1734, the sample libraryoptionally may be augmented by adding the image(s), additional data,facial descriptor measurements, principal indicators, complexindicators, pairwise functions, and/or pairwise complementarityindicator for the pair of individuals.

In other embodiments, the operation in block 1732 may comprise using thepairwise complementarity indicator to assign the pair to one of aplurality, or set, of categories. In some embodiments, a set ofcategories Kt, t=1 to T, are defined by a set of thresholds, Sxt, t=1 toT+1, such that a subject is assigned to category Kt if Sxt<S≤Sx(t+1).Optionally, beginning threshold Sx1 and end threshold Sx(T+1) may be setto −/+∞, respectively. In other embodiments, S may comprise a set ofz-scores, Zq, q=1 to Q—or raw scores, Sq, q=1 to Q—each corresponding topairwise functions f(Pk1, Pk2) and/or f(Cj1, Cj2) of principal and/orcomplex indicators for the two individuals. In such embodiments, each ofthe set of categories Kt, t=1 to T, may be defined by a particular range{Sxqt1, Sxqt2} associated with a particular z-score, Zq. A pair ofindividuals will be assigned to a category Kt if each member of the setof z-scores {Zq} falls within the particular range {Sxqt1, Sxqt2}associated with that z-score and category. In other embodiments, a pairof individuals may be assigned to a particular category if a requiredsubset of the z-scores (or raw scores) falls within the set of rangesassociated with that category. For example, an on-line dating servicecould categorize the type of relationship that two individuals are mostlikely to experience. If the type of relationship that each of theindividuals is seeking was included in the individual additional data,such embodiments could also use it to determine the likelihood ofsuccess of the prospective pairing.

Variations of the embodiment shown in and described above with referenceto FIG. 17 are also possible, as will be apparent to persons of ordinaryskill in the art. For example, rather than predicting whether two newindividuals (i.e., individuals not in the sample library) are pairwisecomplementary, the method of FIG. 17 may also be used to predict whichof the existing members of a sample library would be pairwisecomplementary with a single new individual. Embodiments also may be usedto determine the degree to which a new individual would be pairwisecomplementary with one or more existing members of the sample library,or to identify or rank a predetermined number of existing members of thesample library based on their degree of pairwise complementarity withthe new individual.

In such embodiments involving pairings of a single new individual withone or more existing individuals in the sample library, the operationsin blocks 1720 through 1728 and 1736 of FIG. 17 will be performed onlyfor the new individual, provided that the data records of existingindividuals in the sample library are complete. Furthermore, theoperations in blocks 1730 and 1732 will be performed for each pairing ofthe new individual with a different existing individual in the samplelibrary. The operations of block 1734 may comprise the ranking orordering of the existing individuals tested in blocks 1730 and 1732based on their degree of pairwise complementarity with the newindividual.

Other embodiments of FIGS. 16 and 17 may be used to categorize anexisting relationship of two individuals (e.g., a marriage) in variousways. For example, the pairwise complementarity indicator, S, maycomprise a set of z-scores {Zq}, each corresponding to one or morepairwise functions of principal and/or complex indicators for the twoindividuals. The set of one or more pairwise functions may be selected,as shown in FIG. 16, such that the corresponding Zq represents or isrelated to a specific area in the particular type of relationship ofinterest. Embodiments may categorize each specific area of therelationship by comparing the value of its corresponding z-score Zq toone or more thresholds Sxq, and assigning a quantitative or qualitativecategory accordingly. For example, embodiments may compare Zq to aseries of thresholds {Sxq1, Sxq2, . . . Sxqn}, each representing aboundary between two of the n+1 equal-probability sections of theprobability distribution of the one or more pairwise functionscorresponding to Zq (e.g., quartiles for n=3). Subsequently, the area ofthe relationship corresponding to Zq will be categorized based on whichof the equal-probability sections that Zq falls within. Thiscategorization may be quantitative (e.g., first quartile, sixth decile)or qualitative (e.g., very weak, weak, average, strong, very strong), ora combination thereof. (e.g., low second quartile).

In addition, various embodiments of the present disclosure may be usedto determine whether, or the degree to which, a group of more than twoindividuals are suitable for one or more purposes, tasks, and/orrelationships. For example, embodiments may be used to determine if agroup of individuals is suitable for a specific purpose, task, orrelationship. A group suitability indicator, SG, corresponding to thespecific task may be determined based on linear and/or non-linearcombinations of the individual principal and/or complex indicators, ormay be a statistic of the individual indicators. For example, if the set{C1 i=1 to I} comprises complex indicator C1 for each of the Iindividuals in the group, the group categorical indicator may be basedon mean, median, mode, standard deviation, variance, ratio of standarddeviation to range, average distance between numerically-ordered values,etc. of the values in {C1 i}. The group suitability indicator SG maycomprise a z-score, raw score, percentile score, or other metricindicating whether, or to what extent, the group is suitable for thespecific task. For example, a percentile score may indicate thesuitability ranking of a particular group among all groups engaged inthe same task. The suitability of the group for the task may bedetermined, for example, based on the relationship of SG to a thresholdvalue Sx, as described above with reference to other embodiments.

Similarly, embodiments may be used to determine which one of aparticular set of tasks is most suitable for a particular group ofindividuals. A group categorical indicator, SG, may be determined usingprincipal and/or complex indicators for individuals in the group. Thegroup categorical indicator may be based on linear and/or non-linearcombinations of the principal and/or complex indicators, or may be astatistic of the individual indicators. The group categorical indicatormay further comprise a set of categories {Kt, t=1 to T}, that aredefined by a set of thresholds {Sxt, t=1 to T+1}, such that the group isassigned to category Kt if Sxt<SG≤Sx(t+1). Optionally, beginningthreshold Sx1 and end threshold Sx(T+1) may be set to −/+∞,respectively. Each of the categories Kt may correspond to a particulartask, and the most suitable task for the group may be determined basedon the category Kt to which the group is assigned.

In other embodiments, a set of group suitability indicators {SGt, t=1 toT}, each corresponding to one of the particular set of tasks, may bedetermined based on linear and/or non-linear combinations of theindividual principal and/or complex indicators, or may be a statistic ofthe individual indicators. The set of group indicators {SGt} maycomprise a set of z-scores, raw scores, percentile scores, or othermetrics that indicates whether, or to what extent, the group is suitablefor each of the corresponding set of tasks. For example, a percentilescore for a task t may indicate the suitability ranking of a particulargroup among all groups engaged in task t. The most suitable task for thegroup may be determined, for example, based on selecting the taskcorresponding to the highest group suitability indicator among the set{SGt}.

Other embodiments of the present disclosure may be used to determinewhether, or to what degree, an existing or prospective group ofindividuals fills a predetermined set of roles. The set of roles may bedefined based on what are deemed necessary for a particular task orpurpose, or may be a more generic, or broader, set of roles not orientedor directed toward any particular task or purpose, e.g., “leader”,“facilitator”, “innovator”, “executor”, etc. Suitability indicatormodels can be defined for each role, based on principal and/or complexindividual indicators related to characteristics associated with therespective roles (e.g., sensitivity, dominance, impulsiveness,creativity, etc.). Using these models and the facial descriptormeasurements of the individuals comprising the group, a set ofsuitability indicators {Sri, r=1 to R} corresponding to the set of Rroles may be computed for each individual i in the group. Thesuitability of an individual for a specific role may be determined bycomparing {Sri} to a corresponding threshold to determine a “fit/no fit”rating. Alternately, or in addition, the suitability indicator for theindividual/role combination may comprise degree-of-fit information,which may be categorical, ordinal, integer-valued, real-valued, etc. Ifan individual's indicators { Sri} show that they are suitable formultiple roles, they may be assigned to each suitable role category. Ifindividual's suitability indicators show that they are suitable for noroles, they may be assigned to a “none” role category or to the rolecategory having the greatest degree-of-fit, provided such information isavailable. Alternately, a single categorical indicator model may bedetermined and used to assign the individual to one category based onthe value of their corresponding indicator, in the same manner asdescribed above with respect to group categorical indicators.

The group suitability indicator may be determined in various ways. Itmay be based on some function of the number of individuals in each rolecategory, number or percentage of role categories unoccupied, number orpercentage of individuals in the “none” role category, etc. Alternately,if individual suitability indicators comprise degree-of-fit information,they may be used to determine the group suitability indicator. Forexample, if a particular distribution among role categories is preferred(e.g., for a particular task), the group of individuals can be mapped tothis preferred distribution in a way that maximizes the groupsuitability indicator, which may be computed by a linear (e.g., sum orweighted sum) or non-linear combination of the individual suitabilityindicators corresponding to the individual/role category mapping. Suchmapping could be constrained, for example, by requiring all, a portion,or a specific subset of the role categories to be filled by at least oneindividual.

Other embodiments of the present disclosure may be used to determinewhether, or to what degree, a new individual fits within an existinggroup of individuals for a specific task, purpose, or relationship. Insome embodiments, suitability of the new individual as a member of thegroup may be determined by constraining the roles of the existing groupmembers to remain the same, and then using facial descriptor measurementanalysis techniques to predict whether the new individual will fill adefined missing role within the group. In these embodiments, a set ofsuitability indicators {Sr, r=1 to R} corresponding to the set of Rmissing roles may be computed for the prospective new group member, inthe same manner as described above. The suitability of an individual fora specific role may be determined by comparing {Sr} to a correspondingthreshold to determine a “fit/no fit” rating. Alternately, or inaddition, the set of suitability indicators may comprise degree-of-fitinformation, which may be categorical, ordinal, integer-valued,real-valued, etc. If the new individual's indicators { Sr} show thatthey are suitable for one or more of the missing roles, they may beassigned to each suitable role category.

In other embodiments, the new individual's suitability may be based onthe existence, or degree, of conflict between the predicted role(s) forthe new individual and roles and/or categories of existing groupmembers. The existence, or degree, of conflict may be determined, forexample, based on a pairwise conflict indicator between the newindividual and each of the existing group members. Such an indicator maybe computed in a manner similar to the pairwise complementarityindicators discussed above.

FIG. 18 shows an exemplary method for determining the new individual'ssuitability be based on the existence, or degree, of conflict betweenthe predicted role(s) for the new individual and roles and/or categoriesof existing group members. Although FIG. 18 illustrates the one or moreembodiments by blocks arranged in a specific order, this order is merelyexemplary and the steps or operations comprising the method may beperformed in a different order than shown in the figure. Moreover, aperson of ordinary skill will understand that the blocks shown in FIG.18 may be combined and/or divided into blocks having differentfunctionality.

In block 1800, digital images and additional data are obtained andstored for the new individual, i.e., the prospective group member. Theimages of the new individual are annotated with a plurality of referencepoints and a set of facial descriptor measurements, Fi, i=1 to M, arecomputed based on the annotated reference points. In addition, principalindicators Pk, k=1 to K, are computed using facial descriptormeasurements Fi, and individual complex indicators Cj, j=1 to J, arecomputed using facial descriptor measurements Fi and/or individualprincipal indicators Pk. The models used to compute the principal andcomplex indicators may be determined, for example, as discussed abovewith reference to FIG. 16. Although the operations of block 1800 havebeen described with respect to the new individual, the person ofordinary skill will readily comprehend that they can be repeated, asnecessary, for existing members of the group.

In block 1810, one or more pairwise functions f(Pk, Pki) and/or f(Cj,Cji) of principal and/or complex indicators for the new individual andthe ith existing group member of the group are computed. Each of thefunctions may correspond to a particular area of potential conflictbetween the new individual and an existing group member. In block 1820,the one or more pairwise functions f(Pk, Pki) and/or f(Cj, Cji) are usedto compute a pairwise conflict indicator, S, for the two individuals.The models used to compute the functions and the pairwise conflictindicator may be determined, for example, as discussed above withreference to FIG. 16.

In some embodiments, the pairwise conflict indicator for two individualsmay comprise a set of z-scores {Zq}, each corresponding to one or morepairwise functions of principal and/or complex indicators for the twoindividuals. In some embodiments, the indicator may comprise one or morez-scores corresponding to one or more functions of individual principaland/or complex indicators, such as described above. A set of one or morepairwise functions may be selected, as in FIG. 16, such that thecorresponding Zq corresponds to a specific area of potential conflictbetween the roles and/or relationship of the two individuals. Forexample, pairwise functions may be selected such that certain z-scoresemphasize differences between characteristics of the two individuals,while other pairwise functions may be selected such that other z-scoresemphasize similarities between other characteristics.

In block 1830, it is determined whether any other group members need tobe analyzed for predicting suitability with the prospective new member.If so, the method returns to block 1810. Otherwise, if all members havebeen analyzed, the method proceeds to block 1840 where the suitabilityof the new individual for membership in the existing group is determinedbased on the set of pairwise conflict indicators {Si} corresponding toeach of the existing group members. This may be done in various waysdepending on the particular embodiment. In some embodiments, block 1840may comprise comparing each of the pairwise conflict indicators, Si, toa suitability threshold, Sx, and determining that the new individual issuitable based on some portion of the comparisons having the desiredrelationship (e.g., greater than). The portion may be all, a majority, aparticular number, a particular subset, etc. Put another way, in suchembodiments, the suitability of the new individual may be determinedbased on their pairwise suitability with each of the existing groupmembers.

In other embodiments, particularly ones in which each pairwise conflictindicator Si comprises a set of z-scores {Zqi}, block 1840 may comprisecomputing a function of the collection of z-scores comprising the set ofpairwise conflict indicators{Si}. For example, if higher z-scoresindicate greater likelihood of conflict, the collection of z-scores maybe added to obtain an overall prediction of likelihood of conflictbetween the new member and existing group members. The computed functionmay then be compared against a threshold to determine overallsuitability as a member of the group.

In other embodiments, suitability of the new individual may bedetermined without constraining the roles of existing group members toremain the same. In other words, the suitability of the new individualmay be determined assuming that their presence will cause the roles ofthe existing group members to change (e.g., people move to roles inwhich they are more/most suitable). In this embodiment, the suitabilityof the individual may be determined based on whether their presence willmake the group more suitable for the specific task, purpose, orrelationship. Accordingly, a group suitability indicator for thespecific task may be determined and used, for example, in the mannerdescribed above with respect to other embodiments related to groupsuitability.

In other embodiments, suitability of one or more individuals withrespect to a group may be determined based on whether their presence inthe group is likely to negatively impact the group's performance, e.g.,by the individual filling a negative role. For example, this embodimentmay be applied to an existing group by computing a group suitabilityindicator S for the current existing group, computing additional groupsuitability indicators Si, i=1 to n, for the existing group minus eachof the n members, then comparing each Si to S to see if the deletion ofa particular member causes the group suitability to increase. Similarly,this embodiment may be used to a prospective member of an existing groupby computing a suitability indicator for the current existing group,computing a suitability indicator for the group including theprospective member, then comparing the two suitability indicators to seeif the new member is likely to negatively impact the existing group'sperformance.

Other embodiments may be used to determine strategies for a group ofindividuals, as illustrated by the flowchart of FIG. 19. In someembodiments, the method shown in FIG. 19 may be used to determinestrategies for a group of individuals based on facial descriptormeasurements for individual members of the group. In other embodiments,the method of FIG. 19 may be used to determine a strategy for dealingwith a group of humans in a common situation, e.g., a group ofindividuals in a workplace, a sports team, etc. Although FIG. 19illustrates the one or more embodiments by blocks arranged in a specificorder, this order is merely exemplary and the steps or operationscomprising the method may be performed in a different order than shownin the figure. Moreover, a person of ordinary skill will understand thatthe blocks shown in FIG. 19 may be combined and/or divided into blockshaving different functionality.

In block 1900, one or more optimal combinations of facial descriptormeasurements are selected for predicting one or more characteristics ofinterest, such as principal indicators, complex indicator, and/orsuitability indicators. Block 1900 may be carried out according to themethod described above with reference to FIG. 11 or 13, or in other wayswithin the scope of the present embodiment. The one or morecharacteristics of interest preferably are related to the strategiesunder consideration. As discussed above, block 1900 may includegeneration of a sample library comprising digital images, additionaldata, and a plurality of facial descriptor measurements. The facialmeasurements may be—but are not necessarily—similar to the ones shown inand discussed above in reference to FIGS. 1A through 7T. Non-facialphysical measurements may also be used with the method.

In block 1902, digital images and additional data for an individual ofinterest are obtained and stored, in the same manner as described abovewith reference to the sample library (e.g., block 1100 of FIG. 11). Inblock 1904, reference points consistent with those added to the imagesin the sample library are added to the images of the individual ofinterest. In block 1906, facial descriptor measurements consistent withthose in the sample library are calculated for the individual ofinterest. In block 1908, the one or more optimal combinations determinedin block 1900 are applied to the facial descriptor measurements of theindividual to predict the one or more characteristics of interest, in amanner such as described above with reference to FIGS. 12 and 14. Thepredictors may include primary indicators, complex indicators, and/orsuitability indicators. Blocks 1902 through 1908 are repeated for eachindividual of interest. In block 1912, the predicted characteristics ofinterest for the individuals are used to predict one or morecharacteristics for the group of individuals. Finally, in block 1914,the predicted group characteristics and/or predicted individualcharacteristics are used to determine a strategy for the group. Otherfactors and information may be used in either or both of blocks 1912 and1914, including factors related to individuals, factors related to thegroup, and/or factors related to the individual or group environment.

By further example, a group characteristic such as aggression level canbe predicted according to the method illustrated in FIG. 19 as follows.First, the optimal combination of facial descriptor measurementsdetermined in block 1900 can be utilized according to blocks 1902through 1910 to predict the aggressiveness of individuals in the group.Next, the set of predicted aggression levels can be combined optimallyto form a predictor, or principal indicator, of the level of aggressionwithin the group. In other words, the principal indicator of aggressionlevel for the ith individual in the group can be expressed asPi=α1·F1+α2·F2+ . . . + αM·FM, where F1 . . . FM are the selected facialdescriptor measurements and α1 . . . αM are the optimal linear combiningcoefficients. Likewise, if the group comprises N individuals, thecomplex indicator of group aggression level can be expressed asCG=β1·P1+β2·P2+ . . . + βN·PN, where β1 . . . βN are the optimal linearcombining coefficients for the group. Furthermore, in the combined groupmodel, the individuals may be ordered in a variety of ways. For example,the principal indicators of individual aggression levels P1 . . . PN maybe rearranged in descending numerical order P′1 . . . P′N, with P′1corresponding to the most aggressive individual. In other embodiments,the principal indicators of individual aggression levels P1 . . . PN maybe rearranged based on the ordered values of another principal indicator(e.g., dominance). In any case, each optimal coefficient βi may beselected, at least in part, based on the position within the orderedgroup. This would allow, for example, coefficients βi to be selected toemphasize the characteristics of the most aggressive and/or submissiveindividuals in the group.

Group dynamics and group characteristics among a group of individualscan be predicted in various other ways known to persons skilled in theart. For example, methods may be employed that are analogous to theHartree—Fock equation, which is commonly used to predict the behavior ofthe electrons in an atom. Alternately, a computational method based on aNETLOGO model can be used. An advantage of this approach is that factorsother than the behavioral traits predicted from the facial descriptormeasurements can be incorporated. Such factors may include the estimatedactivity level and the resulting number of interaction betweenindividuals in a group, as well as environmental factors.

Furthermore, computational models based on vector spaces may be employedin any of the above methods related to predicting characteristics ofindividuals, pairs of individuals, or groups of individuals.Conceptually, such a computational model would be based on assigning ameasurement (or combination of measurements, such as an indicatordescribed above) to an axis (or vector) in a multi-dimensional space anddetermining a point in this space as an “optimum.” In other words, suchembodiments would identify the continuous range of values for each ofthe facial measurements (or combinations of facial measurements) and thespecific value on each that is the optimum for a specific characteristicof interest. Subsequently, a performance or suitability measure could bedetermined for one or more individuals (or pairs or groups) of interestbased on the distance (e.g., Euclidean distance) between the values ofthe axis variables for the individuals of interest and the correspondingoptimum value previously determined In such cases, a larger distancegenerally indicates lower performance or suitability of the individualof interest.

The various computational methods illustrated by exemplary FIGS. 11through 19 have been described herein as generating predictors of one ormore characteristic, with each predictor comprising a single value suchas a number, a percentile, a z-score, etc. However, persons of ordinaryskill will understand that no model is capable of perfectly predicting acharacteristic of a specific individual, especially if the specificindividual was not part of the data set from which the model wasderived. Consequently, any of the prediction models described herein mayalso comprise prediction intervals used to show a range of valuessurrounding the predicted value. The width of the prediction intervalsfor a particular predictor depends on both the variability of the dataunderlying the model and the degree of confidence required in theprediction (e.g., 95% confidence level). In some embodiments, theconfidence level for a predictor of a particular characteristic may bedetermined, for example, based on the facial measures and environmentalfactors related to that characteristic. For example, predictors thatcomprised measurements of facial features correlating with capacitiesfor self-reflection, learning, and/or flexibility would require largerintervals due to the effect of these traits on the actual characteristicbeing predicted.

Persons of ordinary skill in the art would understand that any of thesecomputational methods may be embodied in various combinations ofhardware and software. For instance, the computations may be carried outby a specialized or general-purpose digital computer, such as a laptop,desktop, tablet, smartphone, workstation, etc. Moreover, this hardwaremay be programmed to carry out such computations in various ways, suchas by programs written in human-readable languages such as C, C++, etc.and compiled into machine-readable code for execution. Alternately, themethods may be expressed in the particular language of a specializedcomputational software package, such as Matlab, which are furtherinterpreted and/or compiled into machine-readable code.

For example, FIG. 9 shows various hardware and software embodiments of asystem according to embodiments of the present disclosure. Varioushardware devices—such as digital camera 904, smartphone 906, cellularphone 908, tablet computer 910, and laptop computer 912—may be used tocapture a digital image of an individual of interest, such as theexemplary horse 902. The image-capturing device may then store the imagein a memory operably connected to a digital computer. This may becarried out in various ways as illustrated in FIG. 9. For example, theimage capturing device may transmit the image through network 916 viawireless access point 914 to digital computer 920, which may be adesktop computer, server, workstation, or the like. Moreover, wirelessaccess point 914 may be a cellular base station, wireless LAN accesspoint, Bluetooth access point, or any other wireless connection known topersons of ordinary skill in the art. Likewise, network 916 may be alocal- or wide-area, public or private network, or any combinationthereof, including an intranet and/or the Internet.

In other embodiments, the image capturing device transfers the captureddigital image to digital computer 920 through a wired connection 922,such as a Universal Serial Bus (USB) connection. In yet otherembodiments, the captured image(s) may be transferred by removing amemory card from the image capturing device and inserting it into memorycard reader 924 of digital computer 920, which may copy the capturedimages to other memory accessible by or operably connected to digitalcomputer 920. Also within the spirit and scope of the presentdisclosure, the image capturing device may transfer the image, viamethods described above or otherwise well known in the art, to devicesother than digital computer 920, such as tablet computer 910. In suchembodiments, further processing according to the methods describe abovewill occur, for example, in tablet computer 910 rather than in digitalcomputer 920. Similarly, the image capturing device may transfer theimage to network storage unit 926 that is accessible via network 916,e.g., cloud storage. Network storage unit 926 may be configured to beaccessible by some or all of the other devices shown in FIG. 9.

In other embodiments, further processing according to the methodsdescribed above also may take place in the image capturing deviceitself. For example, tablet computer 910 may be used to capture imagesof individuals of interest, store the images in memory accessible oroperably connected to it (including, for example, network storage unit926), and then execute one or more software applications embodying oneor more methods described above. Specific measurements or processed datafrom the image capturing device may also be communicated to a centralcomputer or central location.

Moreover, the terms and descriptions used herein are set forth by way ofillustration only and are not meant as limitations. Those skilled in theart will recognize that many variations are possible within the spiritand scope of the disclosure as defined in the following claims, andtheir equivalents, in which all terms are to be understood in theirbroadest possible sense unless otherwise indicated.

What is claimed is:
 1. A method for predicting a characteristic relatingto a type of individual for an individual of the type, comprising:measuring, using a computer, two or more metrics from reference pointson a digital image representing the individual of the type; calculating,using the computer, a resultant using the two or more measured metricsand a predetermined mathematical formula for the individual; predicting,using the computer, a performance characteristic of the individual thatmay be influenced by social interactions using the calculated resultantfrom the predetermined mathematical formula for the individual and dataabout a group of individuals of the same type as the individual and whoshare social connections with the individual; and presenting thepredicted performance characteristic of the individual.
 2. The method ofclaim 1 wherein the data about the group of individuals is calculatedusing the predetermined mathematical formula.
 3. The method of claim 2further comprising: obtaining the digital image representing theindividual; and assigning the reference points on the obtained digitalimage.
 4. A non-transitory computer readable medium comprising a set ofinstructions for determining a social compatibility of an individualwith a group that, when executed by a processor comprising a computingdevice, causes the computing device to: store one or more digital imagesrepresenting an individual in a memory; annotate the one or more digitalimages with a plurality of reference points; compute one or more metricsrelated to the individual using the plurality of reference points usinga predetermined mathematical function; and predict the socialcompatibility of the individual with other individuals in a group ofsocially connected individuals based on the one or more metrics relatedto the individual, indicators of characteristics related to the otherindividuals in the group, and one or more environmental factors.
 5. Thenon-transitory, computer-readable medium of claim 4, further comprisinginstructions that, when executed by the processor, causes the computingdevice to: store the one or more digital images representing theindividual in a memory operably connected to the computing device;annotate the one or more digital images with at least a portion of theplurality of reference points.
 6. The non-transitory, computer-readablemedium of claim 4: wherein the instructions that, when executed by theprocessor, cause the computing device to predict the compatibility ofthe individual comprise instructions that, when executed by theprocessor, cause the computing device to compute one or morecompatibility indicators based on the one or more individual indicators;and further comprising instructions that, when executed by theprocessor, cause the computing device to provide at least a portion ofthe one or more compatibility indicators to an output device.
 7. Thenon-transitory, computer-readable medium of claim 4, wherein predictingthe social compatibility of the individual is further based on one ormore additional metrics than the one or more metrics used to compute theone or more individual indicators.
 8. A system for predicting one ormore socially influenced characteristics of a group of sociallyconnected individuals of the same type, comprising: at least oneprocessor; at least one memory, operably connected to the at least oneprocessor and configured to store data and software code that, whenexecuted by the at least one processor, causes the system to: for eachof the individuals comprising the group, store one or more digitalimages representing the individual in the at least one memory; annotatethe one or more digital images with a plurality of reference points;compute one or more metrics using the plurality of reference pointsusing predetermined mathematical functions; predicting one or morecharacteristics of each of the individuals comprising the group based onthe one or more metrics; and predicting one or more socially influencedperformance characteristics of a group of socially connected individualsbased on the predicted one or more characteristics of each of theindividuals comprising the group and one or more environmental factors.9. The computerized system of claim 8, wherein the at least one memoryfurther comprises software code that, when executed by the at least oneprocessor, causes the system to: for each of the individuals, store theone or more digital images representing the individual in a memoryoperably connected to the digital computer; and annotate the one or moredigital images representing the individual with at least a portion ofthe plurality of reference points.
 10. The computerized system of claim8, wherein the at least one memory further comprises software code that,when executed by the at least one processor, causes the system toprovide data related to the one or more characteristics of the group toan output device.
 11. A non-transitory computer readable mediumcomprising a set of instructions for predicting one or more sociallyinfluenced characteristics of a group of socially connected individualsof the same type that, when executed by a processor comprising acomputing device, causes the computing device to: for each of theindividuals comprising the group, store one or more digital imagesrepresenting the individual in at least one memory; annotate the one ormore digital images with a plurality of reference points; compute one ormore metrics using the plurality of reference points using predeterminedmathematical functions; predicting one or more characteristics of eachof the individuals comprising the group based on the one or moremetrics; and predicting one or more characteristics of a group ofsocially connected individuals based on the predicted one or morecharacteristics of each of the individuals comprising the group and oneor more environmental factors.
 12. The non-transitory, computer-readablemedium of claim 11, further comprising instructions that, when executedby the processor, causes the computing device to: for each of theindividuals, store the one or more digital images representing theindividual in a memory operably connected to the computing device; andannotate the one or more digital images representing the individual withat least a portion of the plurality of reference points.
 13. Thenon-transitory, computer-readable medium of claim 11, further comprisinginstructions that, when executed by the processor, causes the computingdevice to provide data related to the one or more characteristics of thegroup to an output device.